Interactive browser-based semiotic communication system

ABSTRACT

An interactive semiotic communication system is provided. In one embodiment, the interactive semiotic communication system comprises a software engine configured to allow a user to represent and understand a complex system in a semiotic environment, wherein the semiotic environment comprises a multivariable system. In one embodiment, the interactive semiotic communication system comprises a toolset comprising a semiotic ontological framework and accompanying semiotic representations. In one embodiment, the interactive semiotic communication system comprises a graphical user interface, displayed on a computing device with a display, configured to allow the user to access and select an item from the toolset and otherwise interact with the complex system in the semiotic environment. The software engine of the interactive semiotic communication system, in one embodiment, is implemented on a computing device with a processor.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of and claims priority of U.S.patent application Ser. No. 13/910,460, filed Jun. 5, 2013, and U.S.patent application Ser. No. 13/447,622, filed Apr. 16, 2012, now U.S.Pat. No. 8,482,576, the contents of which are hereby incorporated byreference in their entirety.

BACKGROUND

It is difficult to precisely define complexity. Though certainly anoversimplification, one possible view of the concept is that to becomplex is to be intricate or complicated. In practice, when complexityis studied, it is often within the context of a system. Complex systemsare typically rich with opportunities for discovery and analysis. Thisis true at least because such systems often include an elaboratearrangement of interconnected parts that are influenced by multiplevariables such that certain groups of parts collectively demonstrateproperties that are not necessarily apparent in the properties of theindividual parts. Unfortunately, some of the most interesting of themore intricate and complicated patterns often go undetected. Or, ifdetected, such patterns are often not understood well enough to providemeaningful insight. This is unfortunate because such patterns withincomplex systems have the potential to fuel the discovery of particularlyuseful and interesting knowledge.

More of these complex patterns would be detected and understood if therewere an effective and at least somewhat standardized tool fordocumenting and navigating through the components and characteristics ofa complex system. Further, deploying such a tool in a manner thateffectively leverages modern social and computer networking technologieswould enable a collaborative and dynamic learning environment that wouldbe especially conducive to the derivation and discovery of complexpatterns and therefore the derivation and discovery of useful andinteresting knowledge.

The discussion above is merely provided for general backgroundinformation and is not intended for use as an aid in determining thescope of the claimed subject matter.

SUMMARY

An interactive semiotic communication system is provided. In oneembodiment, the interactive semiotic communication system comprises asoftware engine configured to allow a user to represent and understand acomplex system in a semiotic environment, wherein the semioticenvironment comprises a multivariable system. In one embodiment, theinteractive semiotic communication system comprises a toolset comprisinga semiotic ontological framework and accompanying semioticrepresentations. In one embodiment, the interactive semioticcommunication system comprises a graphical user interface, displayed ona computing device with a display, configured to allow the user toaccess and select an item from the toolset and otherwise interact withthe complex system in the semiotic environment. The software engine ofthe interactive semiotic communication system, in one embodiment, isimplemented on a computing device with a processor. These and othervarious features and advantages that characterize the claimedembodiments will become apparent upon reading the following detaileddescription and upon reviewing the associated drawing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer system.

FIG. 2 is a schematic representation of a complex informationinteraction system that encompasses an interactive portal forfacilitating the representation and exploration of complexity.

FIG. 3 is a diagrammatic representation of a screenshot that includesone embodiment of a portion of the VisualY model framework.

FIG. 4 is a diagrammatic representation of a screenshot that includes alevel of the VisualY model framework.

FIG. 5 is a schematic illustration of the VisualY model framework.

FIG. 6 is a diagrammatic representation of a navigation history tool.

FIG. 7 is a diagrammatic representation of a screenshot including analternative view of a portion of the VisualY model framework.

FIG. 8 is a block diagram showing an engine.

FIG. 9 is a screen shot of a multi-variable system for producing userinterface components on a generative basis.

FIG. 10 is an example screen shot demonstrating an alternate view of aportion of the VisualY model framework.

FIG. 11 is a screen shot of a multi-variable system for producing userinterface components on a generative basis.

FIG. 12 is a screen shot of a user interface for selecting and therebyinitiating display of an instantiation of the VisualY language.

FIG. 13 is an illustrative screenshot representation of a limitedportion of an embodiment of the VisualY framework.

FIG. 14 is a static illustration of a dynamic VisualY representations.

FIG. 15 is a static illustration of a dynamic VisualY representations.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

1. Illustrative Operating Environments

Embodiments of the present invention may illustratively be applied inthe context of a computer system. FIG. 1 is a block diagram of anexemplary computer system within which embodiments of the presentinvention may be implemented. The system includes a conventionalpersonal computer 10, including a processing unit 12, a system memory 14and a system bus 16 that couples the system memory 14 to the processingunit 12. System memory 14 includes read only memory (ROM) 18 and randomaccess memory (RAM) 20. A basic input/output system 22 (BIOS),containing the basic routines that help to transfer information betweenelements within the personal computer 10, is stored in ROM 18.

Embodiments of the present invention may be applied in the context ofcomputer systems other than the FIG. 1 personal computer 10. Otherappropriate computer systems include handheld devices, tablet devices,multi-touch input devices, e-reader devices, any other consumerelectronic device, mainframe computers, multi-processor systems, and thelike. Those skilled in the art will also appreciate that embodiments canalso be applied within computer systems wherein tasks are performed byremote processing devices that are linked through a communicationsnetwork (e.g., communication utilizing Internet or web-based softwaresystems). For example, program modules may be located in either local orremote memory storage devices or simultaneously in both local and remotememory storage devices. Similarly, any storage of data associated withembodiments of the present invention may be accomplished utilizingeither local or remote storage devices, or simultaneously utilizing bothlocal and remote storage devices.

Personal computer 10 further includes a hard disc drive 24, a magneticdisc drive 26, and an optical disc drive 30. Magnetic disc drive 26 canillustratively be utilized to read from or to write to a removable disc28. Optical disc drive 30 can illustratively be utilized for readingdata from (or writing data to) optical media, such as a CD-ROM disc 32.The hard disc drive 24, magnetic disc drive 26 and optical disc drive 30are connected to the system bus 16 by a hard disc drive interface 32, amagnetic disc drive interface 34, and an optical disc drive interface36, respectively. The drives and their associated computer-readablemedia provide nonvolatile storage for the personal computer 10. Othertypes of media that are readable by a computer may also be used in theexemplary operation environment.

A number of program modules may be stored in the drives and RAM 20,including an operating system 38, one or more application programs 40,other program modules 42 and program data 44. In particular, one or moreof the other program modules 42 can illustratively be part of aninteractive portal for facilitating the representation and explorationof complexity. For example, the portal may include features consistentwith one or more portal features that are part of embodiments of thepresent invention described below in greater detail with reference toother Figures.

Input devices including a keyboard 63 and a mouse 65 are functionallyconnected to system bus 16 through a serial port interface 46 that iscoupled to system bus 16. Input devices could be connected utilizingother interfaces such as a universal serial bus (USB). Monitor 48 isconnected to the system bus 16 through a video adapter 50. Otherperipheral output devices (e.g., speakers or printers) could also beincluded but have not been illustrated.

The personal computer 10 may operate in a network environment utilizingconnections to one or more remote computers, such as a remote computer52. The remote computer 52 may be a server, a router, a peer device, orother common network node. Remote computer 52 may include many or all ofthe features and elements described in relation to personal computer 10,although only a memory storage device 54 has been illustrated in FIG. 1.The network connections depicted in FIG. 1 include a local area network(LAN) 56 and a wide area network (WAN) 58. Such network environments arecommonplace in the art. The personal computer 10 is connected to the LAN56 through a network interface 60. The personal computer 10 is alsoconnected to WAN 58 and includes a modem 62 for establishingcommunications over the WAN 58. The modem 62, which may be internal orexternal, is connected to the system bus 16 via the serial portinterface 46. These are but examples of network implementations withinthe scope of the present invention.

In a networked environment, program modules depicted relative to thepersonal computer 10, or portions thereof, may be stored in the remotememory storage device 54. For example, application programs 64 may bestored utilizing memory storage device 54 and may include at least aportion of an interactive portal for facilitating the representation andexploration of complexity, embodiments of which will described below ingreater detail below with reference to other Figures. In addition, dataassociated with an application program, such as the interactive portalconsistent with embodiments of the present invention, may illustrativelybe stored within data storage 66. Data storage 66 is associated withmemory storage device 54 (alternatively, or in combination, theapplication data may be stored within one of the data storage locationsassociated with computer 10). It will be appreciated that the networkconnections shown in FIG. 1 are exemplary and other means forestablishing a communications link between the computers, such as awireless interface communications link, may be used.

Notwithstanding the broad applicability of the principles of theinvention described above, it should be understood that theconfiguration of the exemplary embodiment as an application program thatis stored on, and enables data to be stored on, a remote computer suchas remote computer 52 provides significant advantages. For example, sucha configuration minimizes the functional requirements (e.g., storagecapacity) of personal computer(s) 10 and enables a broader range ofpotential adaptive control features, simplified system updating andupgrading features, and also simplified debugging and troubleshootingfeatures.

2. Illustrative Application Architecture

FIG. 2 is a schematic representation of a complex informationinteraction system 100 that encompasses an interactive portal forfacilitating the representation and exploration of complexity, variouscomponents and features of which will be described throughout theremainder of the present description. System 100 includes servercomponents 102, which include one or more databases 106. Databases 106are illustratively utilized within system 100 to store (in whole or inpart) representations 162 of complex systems and a related framework164, embodiments of which will be discussed below in relation to otherFigures. The server components 102 also include a complex informationinteraction application 104 and a set of data management components 108.Data management components 108 are illustratively configured to respondto input commands (e.g., automatically generated and/or human-userinitiated commands) originating from the application 104. For example,components 108 illustratively interact with database 106 so as to managethe storing and retrieving of the representations 162, framework 164,and other related data based on commands from the application 104.Components 108 are also configured to manage any of a variety of otherdatabase management functions such as, but not limited to, data accessmanagement and functions related to maintaining data security.

Those skilled in the art will appreciate that application 104, datamanagement components 108 and database 106 need not all be, asillustrated, in the same server environment or even in the samelocation. They could just as easily be distributed across multipleserver environments and/or multiple locations without departing from thescope of the present invention. All possible combinations ofdistribution are contemplated and are to be considered within the scopeof the present invention.

Complex information interaction application 104 is illustratively anetwork-based application that includes a public web interfacemaintained on the Internet or some other computer network. In oneembodiment, application 104 is coded in a browser-supported language.For example, not by limitation, application 104 can be coded inJavaScript in combination with a browser-rendered markup language suchas, but not limited to, HTML. The code might also encompass style sheetsas a means for influencing the presentation of the application to theuser. Those skilled in the art will appreciate that this is but one ofmany examples of how a web-based application can be implemented withoutdeparting from the scope of the present invention. Otherimplementations, including any coding combination implementing any webapplication development tools are also within the scope. Regardless ofthe precise implementation, users are illustratively able to interactwith application 104 remotely utilizing any of a wide variety ofdifferent computing devices. Devices within the scope of the presentinvention include, but are certainly not limited to, a mobile phone, alaptop PC, a desktop PC, a mobile device, a personal digital assistant,a multi-touch input device, an e-reader device, a tablet PC, a portablemusic player, or any other similar electronic device.

Accordingly, the user-focused front end of application 104 is, in oneembodiment, reliant upon a client side web browser for ultimateexecution, including presentation to and interaction with a user of thebrowser/application. For this reason, browser applications 107, 109, 105and 111 are shown in FIG. 2 as supporting access to application 104 byusers 110, 112, 114 and 120, respectively. As will become apparent,users 110, 112 and 114 (and probably also the gatekeeper user 120) areable to utilize application 104 in combination with their browserapplication to experience (e.g., view, listen to, navigate through,etc.) the complex system representations 162 (and the related framework164) stored, in whole or in part, in database 106. Further, users 110,112 and 114 (and probably also the gatekeeper user 120) can also, ifthey wish, utilize application 104 in combination with their browserapplication to make contributions to and edit the complex systemrepresentations 162 (and related contextual portions of framework 164)stored, in whole or in part, in database 106. As will also becomeapparent, gatekeeper 120 manages, to some extent, modifications andadditions to the representations 162 (and related contextual portions offramework 164) stored, in whole or in part, in database 106.

For the purpose of simplifying the description of the present invention,only the representative set of users 110, 112, 114 and 120 are shown asremotely accessing application 104 over a network 103, which isillustratively but not necessarily the Internet. Further, it should beunderstood that it is contemplated that different users could, intheory, access application 104 over different computer networks ratherthan, as is illustrated in FIG. 2, over the same network. Further, oneor more users could access application 104 by means other than over apublic or other computer network. For example, user 120 is shown in FIG.2 as having both network-based and more direct access to application104. User 120 may, in one embodiment, also have network based or moredirect access to either or both of components 106 and 108. Anycombination of computer-implemented user access connection/communicationschemes from accessing the server components 102 should be consideredwithin the scope of the present invention. Any user can be provided withpublic remote and/or more direct access to the server-based componentsof system 100 (e.g., components 102) without departing from the scope ofthe present invention.

Gatekeeper 120 is illustratively a system administrator having broaderaccess rights and privileges than users 110, 112 and 114. Gatekeeper 120controls, at least to some degree, the configuration and functionalityof components 104, 106 and 108, and therefore has the ability to affectthe application 104 experience from the perspective of users 110, 112and 114. In one embodiment, the gatekeeper 120 sets and/or enforcesconditions under which the complex system representations 162 andframework 164 may be modified or expanded upon by users 110, 112 and114. The described network architecture enables gatekeeper 120 to makesystem adjustments and modifications without having to create anddistribute frequent software updates to the users. Instead, thegatekeeper is able to make changes quickly and, if desired, frequently.Of course, it is contemplated that there may be multiple or even manygatekeepers working independently or cooperatively to maintain andmanage the database 106 content and/or its related functionalcomponents.

Those skilled will appreciate that there are many specificimplementation options for the physical and functional configuration ofdatabase 106 and its related components 108. Each option has its own setof drawbacks and benefits. The best option will depend at least uponavailable resources both in terms of equipment and personnel. For thepurpose of providing a complete and thorough description of embodimentsof the present invention, it is worth expounding upon a few different ofthe many possibilities.

In one embodiment, at least part of database 106 is implemented as arelational database comprising a collection of tables that storeparticular sets of data pertinent to the stored representations 162,framework 164 and other related data. However, without departing fromthe scope of the present invention, database 106 can include any othertype of database technology such as but certainly not limited to anobject database, an object-relational database, or a native xmldatabase. All of these and any other database implementation should beconsidered within the scope of the present invention.

Those skilled in the art will appreciate that the configuration andfunctionality of components 108 will vary as necessary to accommodatewhichever type of database technology is included as part of theimplementation of database 106. For example, in a scenario in whichdatabase 106 includes a SQL database implementation, such as MySQL orSQL Server offered by Microsoft Corporation of Redmond, Wash., then thedata management components 108 are likely to include a PHP or Pythonaccess/security layer. In an object-relational (O-R) database scenario,components 108 illustratively include support for referring to storeddata in terms of objects and their properties rather than the morespecific elements of a more traditional database schema (e.g., atable-based schema). For example, in order to support the O-R system,components 108 are likely to include a mapping mechanism (e.g., aclass-table mapping mechanism) that facilitates a translation ofobject-oriented database calls into corresponding relational databasecalls.

In still another embodiment, database 106 includes an implementation ofNoSQL, such as a MongoDB database implementation. In this case, thedatabase will generally not be organized in tables similar to arelational database but instead will be organized around flat stackobjects. One advantage of a NoSQL implementation in the context of thepresent invention is that it allows for especially efficient access towhat are likely to be particularly large data sets. This is truebecause, generally speaking, a NoSQL implementation requires fewerserver resources than, for example, a relational database. Of course, toaccommodate a NoSQL implementation, appropriate data managementcomponents 108 will be cooperatively implemented. For example, whendatabase 106 includes a NoSQL implementation, components 108illustratively include a Python, or Ruby, or PHP access/security layer.Of course, those skilled in the art will appreciate that these are butexamples of possibilities within the scope of the present invention.

Those skilled in the art will also appreciate that components 106/108are configured to support any of a plurality of different data formatand exchange protocols that enable communication between application 104and the remote computers of the users 110, 112, 114 and 120. In oneembodiment, the protocol is such that it supports a pulling of live codefrom the database for immediate execution in conjunction withapplication 104 and/or in conjunction with an application (e.g., thebrowser application) operating on one of the remote user computers. Thiscapability is particularly well suited for data generative functions ofapplication 104 contemplated in accordance with embodiments of thepresent invention that will be described below in greater detail inrelation to other Figures.

One option for the data format and exchange protocol is XML. However, inone embodiment, application 104 is an Ajax-style web application thatsupports out-of-band, lightweight calls to database 106. These calls areillustratively initiated using JavaScript and involve formatting data,sending it to the web server, and parsing and working with the returneddata. While most browsers can construct, send, and parse XML, JavaScriptObject Notation (or JSON) provides a standardized data exchange formatthat is better-suited for Ajax-style web applications. For example, theparsing required in an XML implementation is simplified or eliminated inthe context of a JSON implementation. Thus, in one embodiment, but notby limitation, JSON is utilized as the data format and exchange protocolat least in, but not limited to, embodiments where the application 104involves a JavaScript implementation in the context of an Ajax webapplication.

In terms of the particular configuration of components 108 and database106, the scope of the present invention is not limited to any oneparticular setup in conjunction with a JSON implementation. For example,in one embodiment, database 106 includes a MongoDB implementation inconjunctions with a Python or Ruby (or any other suitableimplementation) in the access/security layer. However, another optionwithin the scope of the present invention is a SQL-type database 106implementation in conjunction with a PHP access/security layerconfigured to facilitate a conversion into JSON-type stacks. These arebut two examples of JSON implementations within the scope of the presentinvention. The scope is not limited to these or any other particularimplementation.

In accordance with one embodiment of the present invention, at least oneof the representations 162 (and its associated portions of framework164) accessed by way of application 104 is graphically presented (e.g.,presented in the context of a portion of framework 164) within theuser's browser based on an implementation of Web-based Graphics Language(WebGL) technology. Those skilled in the art will appreciate that WebGLgenerally refers to a technology involving utilization of JavaScript soas to implement 3D graphics within a user's browser. More particularly,WebGL is a context of the canvas HTML element that provides a 3Dgraphics API implemented in a web browser without the use of a plug-in.Of course, the WebGL functionality requires the user to operate a WebGLcompatible browser. However, it is believed by many in the art that mostbrowsers in the future will support WebGL graphics technology.Compatible browser applications are presently readily available on theInternet and otherwise. It should be noted that the scope of the presentinvention is not limited to WebGL technology specifically. Other similartechnologies (e.g., similar technology offered by Microsoft Corporationof Redmond, Wash.) can be included in the system integration withoutdeparting from the scope of the present invention.

Audio within web applications is currently most frequently deliveredthrough plugins such as Flash and QuickTime. This may be true in thecontext of system 100. However, in accordance with one aspect of thepresent invention, at least one of the representations 162 accessed byway of application 104 includes audio presented within the user'sbrowser based on an implementation of a JavaScript API for processingand synthesizing audio. Accordingly, in one embodiment, audio issynthesized and processed directly in JavaScript without use of aplug-in. In one embodiment, audio is managed through the AudioData API(Mozila Firefix) and/or the WebAudio API (Chrome and Sarari) therebyenabling audio information to be efficiently manipulated directly or atleast relatively more directly upon a user's hardware responsible foraudio data management. In one embodiment, the audio is implementedwithin the applications such that the representation includes the audioused with graphics generated using the canvas 2D and/or WebGL 3Dgraphics APIs.

3. The VisualY Language

Words that humans think, speak and write are mostly based upon theincremental development of language instead of any purposefully inventedsystem of communication. In most cases, the origin of these words datesback hundreds and even thousands of years—perhaps 50 to 100 generations.However, it has been only within about the past 400 years (spanning just10 or so generations) that an understanding of the science of our worldand existence entered a degree of accuracy and certainty that couldreasonably serve as a workable outward foundation of accurate knowledgeand thought.

Under the circumstances, language has developed into something otherthan a scientifically founded invention having as its objective being amost effective means of organizing and communicating information andknowledge. Instead, language has become a significantly limited andgenerally imprecise accumulation of human constructs that onceestablished became very difficult to modify or discard. And becausehumans generally think within the confines of language, they thereforefind it very difficult to work outside of it. It is therefore bothreasonable and logical for the continued advancement of humanunderstanding that current language-driven systems for organizing andcommunicating information and knowledge be improved or replaced.

Application 104, in accordance with one aspect of the present invention,is implemented in a manner the same (or similar) as described inrelation to FIG. 2 but is done so as part of a scheme for overcoming thenoted and other shortcomings of current systems of language. Moreparticularly, application 104 is programmed to provide an advancedcomputer-implemented system for managing and presenting information andknowledge in a way that more accurately addresses how the components ofthe world and universe are actually organized—in particular assynchronous and asynchronous complex systems. In contrast, currentsystems of language are frustratingly linear in that they are largelydependent upon words that in many cases are only capable of expressing avery limited scale and scope of complexity.

In one embodiment, a key function of application 104 is to facilitaterepresentation and communication of information and knowledge inaccordance with a dynamic interactive visual and auditory language,which will be referred to herein as the language of VisualY. As willbecome apparent, an instantiation of the VisualY language will mostoften be presented as a visual and/or auditory representation ofinformation/knowledge set against the context of all or a portion of amulti-level ontological system of maps and map views. The system of mapand map views is illustratively the framework 164 in FIG. 2. Theframework 164 may be referred to herein as the multi-level ontologicalmodel framework 164, as model framework 164, or as the VisualY modelframework 164. The information/knowledge representations areillustratively the representations 162 in FIG. 2. The representations162 may be referred to herein as the VisualY representations 162.

Application 104 utilizes a management component 166 to facilitate thecreation, organization, collection, modification, transformation,presentation, navigation, browsing, searching, experiencing, displaying,storage, retrieving, transference, and otherwise communication ormanipulation of instantiations of the VisualY language. Accordingly,application 104 utilizes management component 166 to facilitate thecreation, organization, collection, modification, transformation,presentation, navigation, browsing, searching, experiencing, displaying,storage, retrieving, transference, and otherwise communication ormanipulation of the VisualY representations 162, most often presented inthe context of all or a portion of the VisualY model framework 164.While the representations 162 and the model framework 164 are beingpresented as elements separately stored in database 106, the linebetween these two components in actual implementation is not likely tobe so distinct. Accordingly, it is likely also true that application 104utilizes management component 166 to facilitate the creation,organization, collection, modification, transformation, presentation,navigation, browsing, searching, experiencing, displaying, storage,retrieving, transference, and otherwise communication or manipulation ofthe VisualY model framework 164.

Application 104 also includes a set of user-driven development tools168. Tools 168, which will illustratively be referred to herein as theVisualY tools or collectively as the VisualY toolset, are utilized by auser (e.g., by any or all of users 110, 112, 114 and 120 interactingwith application 104 by way of a web browser) to interact withinstantiations of the VisualY language as described in the paragraphbefore this one. This means that the VisualY toolset 168 may be utilizedby the users to create and/or modify VisualY representations 162 and/orthe VisualY model framework 164. In one embodiment, it is within thescope of the present invention for each user to each create their ownVisualY instantiations and utilize functions included within application104 to share access and/or modification rights as desired with oneanother. However, in another embodiment, the application is configuredto enable all of the users to access and/or modify the same VisualYinstantiations.

Thus, in one embodiment, application 104 operates in a manner similar tothe known Wikipedia system currently made available on the WorldWideWebat www.wikipedia.org. For example, application 104 illustrativelyenables the users to selectively access VisualY instantiations but theusers are also able to use the VisualY toolkit 168 to collectively, asthe community of users of application 104, add to, delete or otherwisemodify the VisualY instantiations or at least portions thereof. Thisillustratively includes the collaborative adding to, deleting and/ormodifying of the VisualY model framework 164 and/or the VisualYrepresentations 162.

In one embodiment, one or more gatekeepers, an example of which isgatekeeper 120, are provided with access to administrative functionsthat enable them to approve or reject changes made by the broader usercommunity to one or more instantiations of the VisualY language. Thoseskilled in the art will appreciate that this authority to accept orreject changes can be manifested in many different ways. For example, inone embodiment, no changes are allowed without approval by a gatekeeper.In other embodiment, changes made by some users are allowed withoutapproval but changes made by other users require approval. Further,different users may be assigned different access rights in terms of whatspecific types of changes they are allowed to make to VisualYinstantiations. These rights may vary from one instantiation to the nextor may be consistent across an entire collection of instantiations.

In response to user-initiated command interactions with application 104,a VisualY instantiation (e.g., a combination of a representation 162 anda corresponding model framework 164 context) is retrieved by component166 and, in one embodiment, all or a portion of it is presented to auser (e.g., displayed on a display such as but not limited to display 48in FIG. 1, and/or provided in part as audio, is provided in aconveniently printable format, etc.). In one embodiment, this processincludes data management components 108 receiving from component 166 arequest (e.g., a user or application initiated query originating fromone of browser applications 105, 107, 109 or 113) that specifies, eitherdirectly or indirectly, all or a portion of the VisualY instantiation tobe retrieved from database 106. The requested representation/modelframework data is retrieved from the database 106 and returned tocomponents 108 in the form of database results. In one embodiment,components 108 include a data formatting component that formulates thedatabase results into a proper format (e.g., a format identified in therequest, identified within the retrieved data or otherwise identified inthe code of application 104), such as in the form of one or more objectsor simply as a data set. In any case, the properly formatted data isprovided as an output to application 104 in response to the applicationrequest. The results are transmitted over network 103 and presented tothe user (e.g., the requesting user) (e.g., one or more of users 110,112, 114 or 120) through their browser application, which facilitatesexecution of software code that is part of application 104, though it iswithin the scope for executed application code to originate from othersources such as but not limited to being embedded within data stored inthe database 106.

FIG. 3 is a diagrammatic representation of a screenshot 300 thatincludes one embodiment of a portion of the VisualY model framework 164(i.e., the multi-level ontological model framework 164). The portionshown in FIG. 3 represents but one level of the model framework. Line306 divides the illustrated level into left section 308 and rightsection 310. Each of sections 308 and 310 is divided into a collectionof cells 312, only an exemplary set of which have been included in orderto simplify the illustration. Situated at the bottom of the screenshotis a plurality of buttons 314-322 that are not technically part of themodel framework but instead are controls for performing functions inrelation to a representation 162 depicted in relation to a portion of adisplayed model framework level, or for transitioning betweenrepresentations 162 and/or portions of the model framework 164.

Before proceeding with the description of screenshot 300 and itsillustrated portion of the model framework 164, it is first worth notingthat, without departing from the scope of the present invention, theconfiguration and shape of cells 312 may be very different than what isshown in any Figure included herewith. For example, straight lines mightbe replaced with non-straight lines, or there could be more or fewercells 312, etc. The stylistic design choices reflected in the Figuresprovided herewith are intended to be exemplary only and are in no way tobe construed as limiting the scope of the present invention.

In one embodiment, some or all of cells 312 are user-selectable (e.g.,but not by limitation, by way of a mouse click, voice recognition, touchscreen input, gesture recognition, etc.) so as to cause navigation(e.g., a transition of the display/screenshot) to a different level ofthe model framework 164. When a symbol appears within a cell 312, thesymbol illustratively is representative of the subject matter and/orconfigurations of other cells 312 obtainable through thesymbol-containing cell. For example, in one embodiment, not bylimitation, the cell 312 that includes symbol 324 indicates to the userthat there are at least four additional cells 312 that can be navigatedto by selecting the cell 312 (i.e., one symbol for each of theadditional cells 312). Similarly, the cell 312 that includes symbol 326indicates to the user that there are at least two additional cells 312(i.e., represented by the two symbols) that can be navigated to byselecting the cell 312 that encompasses the two symbols. This is not tosay that there must always be a symbol for every cell 312 reachablethrough the symbol-containing cell 312. The numbers of symbols includedin a given cell 312, for example, may be dependent upon the currentdisplayed size of the given cell. In one embodiment, the application isconfigured to change the size of the symbols or even change the symbolsthemselves depending upon cell size. In one embodiment, a short-handsymbol (e.g., a dot) is utilized to show existence of a particularsymbol when the symbol-containing cell is not large enough to displaythe particular symbol in a visually appealing way.

It should be noted that the described symbol-based scheme is only anexample of how selectability/navigability might be indicated. Forexample, but not by limitation, the indication could just as easily be amore unified symbol such as a shape containing a dot for each cell 312in the next selectable level of the model framework. Further, theindication need not necessarily include any symbol per se. It is withinthe scope of the present invention for the indication to be a letter, anumber, a plus sign, any other character, a shape that is not a letteror number, a color, cell shape, cell location, cell location in relationto a separate indication of selectability, a sound, a change in color, achange in sound, a change in characters, or even an animation. Any ofthese or any other possibility can be utilized to indicate that aparticular cell 312 is selectable/navigable to a different level of themodel framework 164.

It is also to be understood that an indication ofselectability/navigability may be selectively displayed such as based onuser preferences or interactions. For example, in one embodiment, anindication is only activated and provided when the user has expressed aninterest in the associated cell 312 (e.g., for example, but not bylimitation, by “mousing over” the cell). Any process for temporarily orconditionally presenting one or more of the indications should beconsidered within the scope of the present invention.

The symbols included within a cell 312 may or may not be an indicationor reminder of a type of subject matter pertinent to the cell and/orpertinent to corresponding lower levels of the model framework 164accessible via selection/navigation of the cell. For example, a plantsymbol might indicate that components of the model framework 164accessed through the cell containing the plant symbol pertain toplant-oriented subject matter. It should be noted that an indication ofsubject matter relevancy need not necessarily be a symbol per se. It iswithin the scope of the present invention for the indication to be aletter, a number, any other character, a color, cell shape, celllocation, a sound, a change in color, a change in sound, a change incharacters, or even an animation. Any of these or any other possibilitycan be utilized to indicate that a given cell 312 pertains to aparticular category of subject matter. It is also to be understood thatthe subject matter indication need not always be displayed for a givencell. In one embodiment, the indication is only displayed when the userhas expressed an interest in the associated cell 312 (e.g., for example,but not by limitation, by “mousing over” the cell). Any process fortemporarily or conditionally displaying one or more of the subjectmatter indications should be considered within the scope of the presentinvention.

As has been described, the screenshot of FIG. 3 represents only aportion of the VisualY framework 164, namely one level of the framework.FIG. 4 is a diagrammatic representation of a screenshot 400 displayinganother level included in the framework. The level shown in FIG. 4includes its own collection of cells 312. The plurality of buttons314-322 are again situated at the bottom of the screenshot in FIG. 4.Again, these buttons are generally not part of the model framework butinstead are controls for performing functions in relation to a currentlypresented VisualY instantiation.

The FIG. 4 screenshot is illustratively what the user sees uponselection of the cell 312 in FIG. 4 that includes symbol 324. In otherwords, the user navigates from the level of the model framework shown inFIG. 3 to the level shown in FIG. 4 by selecting the cell 312 containingsymbol 324. Notably, the cell in the FIG. 3 screenshot that includessymbol 324 is one of four symbols in the cell. The same four symbolsappear again in the FIG. 4 screenshot but this time as labels for a setof different cells 312 that include new symbols. The new cells in theFIG. 4 screen shot can illustratively be selected in order to navigateto still another level of the model framework 164. A similar butdifferent set of symbols is utilized in the FIG. 4 cells 312 to againindicate navigability options and/or subject matter relevancy.

FIG. 5 is a broader schematic illustration of an embodiment of amulti-level ontological model framework 164, illustratively the sameframework reflected in the screenshots of FIGS. 3 and 4. This schematicillustration is intended to make clearer the hierarchically navigablenature of the model framework 164. Each of a plurality model segments502 (only a representative few have been specifically labeled in FIG. 5)illustratively corresponds to a cell 312 included in the framework 164.For example, the model segments 502 on a top level 506 illustrativelycorrespond to the cells 312 included in the level of the model frameworkshown in the screenshot of FIG. 3. The model segment furthest to theleft on level 506 illustratively corresponds to the cell in FIG. 3 thatincludes symbol 324. The arrow 552 in FIG. 5 pointing from level 506 tolevel 508 indicates that the user is provided with the ability, ifdesired, to select the symbol 324 cell so as to initiate a transition tothe four pointed-at model segments 502 in level 508. These four modelsegments in 508 illustratively correspond to the cells 312 shown in FIG.4 such that the transition between levels 506 and 508 illustrativelyincludes a transition of the user interface, as was described inrelation to FIGS. 3 and 4.

In addition to levels 506 and 508, the illustration of FIG. 5 alsoincludes levels 510 and 512. Each level includes its own model segments502, which illustratively correspond to cells 312 included in otherlevels of the framework 164, and therefore in other screenshotsavailable for display depending user-initiated transitions betweenlevels. Arrows between levels are utilized to represent opportunities tonavigate to a particular sub-set of cells 312 that correspond to asub-set of model segments 502 on a given level.

It is to be noted that the illustration of FIG. 5 does not exactlycorrespond to the example screenshots shown in FIGS. 3 and 4. Theillustration is simplified at least in that an actual implementationvery well may include many more model segments 502 corresponding to manymore cells 312 included within the model framework 164. Further, anactual implementation very well may include many more levels than justthe four levels shown in FIG. 5. Still further, the navigation paths(e.g., schematically represented in FIG. 5 by arrow 552 and the othersimilar arrows) very well may be more complicated than a simpleone-to-several linking of model segments 502. In one embodiment,multiple model segments 502 are linked to the same group of modelsegments 502 indicating that more than one cell 312 can beselected/navigated to the same screenshot view of a given sub-set ofcells 312 within the model framework 164. Those skilled in the art willappreciate the many ways, all to be considered within the scope of thepresent invention, in which the model framework 164 is able to evolve soas to become more complicated than the illustrated scenario. Theillustrated scenario has been intentionally simplified in order to makeclear the broad nature of various embodiments of the present invention.The navigation paths incorporated into an actual implementation of modelframework 164 can skip levels of the hierarchy up or down, or canotherwise be configured in any manner without departing from the scopeof the present invention.

In accordance with one aspect of the present invention, the system isconfigured to support navigation from a first level within the modelframework 164 to a second level even when there is not a cell 312 in thefirst level that supports direct navigation to the second level. Inother words, the user is provided with options for navigating betweensub-sets of cells 312 without necessarily following a “connected” pathbetween levels of the framework (e.g., levels 506, 508, 510 and 512).

FIG. 6 is a diagrammatic representation of a navigation history tool600. Tool 600 is an example of one embodiment of how a non-connectednavigation path between levels of the framework 164 can be implementedas part of application 104. Component 600 is illustratively rendered onthe application display so as to provide a visual indication of a recent(e.g., most recent) navigation path taken through framework 164 in orderto arrive at a currently displayed subset of cells 312. Component 600includes four boxes chronologically arranged from left to right suchthat the currently viewed level corresponds to the box furthest to theright (i.e., box 608) and the first viewed level corresponds to the boxfurthest to the left (i.e., box 602). Of course, the two boxes in themiddle (i.e., boxes 604 and 606) represent consecutive intermediatelyviewed levels along the navigation path from the first viewed sub-set ofcells 312 to the currently viewed sub-set of cells 312.

The symbols utilized in each of the boxes of component 600 areillustratively indicative (e.g., indicative of subject matter, etc) ofthe corresponding sub-set of cells 312. For example, the symbol shown inbox 602 reflects the fact that the user began at the sub-set of cells312 reflected in the screenshot 300 of FIG. 3. The triangular symbolshown in box 604 reflects the fact that the user selected a cell 312that caused navigation to a second sub-set of cells 312 (not illustratedin the Figures of the present description). The oval-shaped symbol shownin box 606 reflects the fact that the user selected a cell 312 thatcaused navigation to a third sub-set of cells 312 (not illustrated inthe Figures of the present description). Finally, the symbol shown inbox 608 reflects the fact that the user selected a cell 312 that causednavigation to a currently displayed sub-set of cells 312 (notillustrated in the Figures of the present description).

Component 600 is illustratively configured to support navigation fromthe screenshot view of cells 312 associated with box 608 to thescreenshot view of cells 312 associated with box 602 without visitingthe screenshots view of cells 312 associated with either of boxes 604 of606. The user is essentially able to “jump back” to the screenshot ofbox 602 (i.e., the view of FIG. 3) by simply selecting (e.g., clickingon, voice activating, touch screen input selecting, gesture detectioninputting, etc.) box 602 from user interface component 600. Thus, theuser is not forced to back out through the previously visited screenshotviews associated with the intermediate boxes 604 and 606 to obtain thepreviously visited view. The user can, of course, assumedly utilizecomponent 600 to jump back to any screenshot view indicated in the userinterface component 600 regardless of how many previously visitedsub-sets of cells 312 are included in the record of navigation history.This is but one example of how the hierarchy of the model framework 304can be navigated through without being strictly bound to a hierarchicalsequence.

Another aspect of the present invention pertains to still anotherembodiment of how application 104 is configured to support navigationbetween non-connected collections of cells 312. FIG. 7 is a diagrammaticrepresentation of a screenshot 700. Screenshot 700 is illustratively analternative view of portions of the multi-level ontological modelframework 164. For example, in one embodiment, a user of application 104has the option of choosing the broader and more encompassing frameworkdisplay scheme of FIG. 7 instead of (or, as will become apparent, incombination with) the framework display scheme described in relation toFIGS. 3 and 4. While those skilled in the art will appreciate that thereare many user input control options for effectuating a transitionbetween different framework display schemes, in one particularembodiment the transition is initiated at least semi-automatically whenthe user inputs a particular framework navigation command that triggersthe transition even though the user may not entirely expect thetransition to be a triggered by the particular command. Alternatively,the transition may be initiated more intentionally, for example, when auser chooses to activate a scheme selection mechanism made accessible byway of one of buttons 314-322 or otherwise.

The framework display scheme of FIG. 7 differs from the scheme of FIGS.3 and 4 in that it represents much more, if not all of, the modelframework in a single screen view. In accordance with the FIG. 3/4scheme, the user navigates from one cell 312 collection to another, forexample, from a cell cluster on one level to a cell cluster on anotherlevel or between two cell clusters on the same level, such that one cellcluster is shown at a time. However, it should be noted that it iscertainly within the scope of the present invention for the FIG. 3/4scheme to be adapted to support renderings of multiple cell clustersfrom different framework levels (or even on the same framework level) tobe simultaneously rendered and presented to the user in the samescreenshot. For example, the framework and related navigation schemecould just as easily be configured such that screenshot 400 of FIG. 4includes multiple separate sets of cells 312 from the same level or fromdifferent levels. It is worth noting that the generally two-dimensionalscheme of FIGS. 3/4 make it difficult to depict relationships betweencell clusters, especially between cell clusters from different levels.This is disadvantage because certain VisualY representations 162 willutilize color, sound, motion or other information representingmechanisms to show, in the context of the framework 164, relationshipsbetween cell clusters and components of cell clusters.

In contrast, the framework display scheme of FIG. 7 enables asimultaneous presentation of multiple cell 312 collections from multipleor all levels of the framework (e.g., levels 506, 508, 510 and 512 asshown in FIG. 5). The scheme is even extensible to show multiple cellclusters from the same framework level. In one embodiment, thescreenshot 700 of FIG. 7 is configured such that a user can select(e.g., by way of voice, mouse, touch, gesture or any other input meansdescribed herein or otherwise) any of the individual collection of cells312. This type of selection illustratively causes a direct transition toa different, corresponding display configuration. For example, in oneembodiment, selecting item 708 from top level 506 illustratively causesthe display to transition to a framework 164 view consistent withscreenshot 300 shown in FIG. 3. In another example, selecting item 706from level 508 illustratively causes the display to transition to aframework 164 view consistent with screenshot 400 shown in FIG. 4. Thisis not to say that the scope of the present invention is limited totransitioning to a “flat” depiction of cell clusters similar to the FIG.3/4 presentation scheme. In one embodiment, a cell cluster presentationto which navigation occurs from a component of the FIG. 7 representationis also relational in that it includes depictions of other relativelyclosely related model framework components. In one embodiment, therelated components are greyed out or otherwise de-emphasized in order tohighlight a particular cell cluster as the cluster to which the user hasnavigated relative to the components that are only related to theparticular cell cluster. Of course, a mechanism for transitioning backto the “bigger picture” framework view of FIG. 7 is likely to beincluded as a user-selectable option. This could be but is not limitedto a mechanism similar to that shown in FIG. 6 and/or a mechanism madeaccessible by way of one of buttons 314-322.

In another embodiment, the framework display scheme consistent withscreenshot 700 of FIG. 7 is configured to enable user selection (e.g.,by way of voice, mouse, touch, gesture or any other input meansdescribed herein or otherwise) of any of the open space 710 (only someexamples of such space are identified in FIG. 7) on a given frameworklevel. This type of selection illustratively causes a direct transitionto a corresponding display of multiple or all sub-sets of cells 312 onthe selected level (as compared to screenshot 400 in FIG. 4, which onlyshows one sub-set of cells 312). Each sub-set of cells 312illustratively is configured to behave and enable subsequent navigationin a manner similar to that described in relation to the single sub-setof cells 312 shown in screenshot 400 of FIG. 4. Of course, in oneembodiment, the system is also configured to enable a transfer from adisplay of multiple cell 312 sub-sets to a display of a single cell 312sub-set that looks and behaves similar to the cell 312 sub-set shown inFIG. 4. Again, a mechanism for transitioning back to a previouslydisplayed framework view is likely to be included as a user-selectableoption. This could be, but is not limited to, a mechanism similar tothat shown in FIG. 6 and/or a mechanism made accessible by way of one ofbuttons 314-322.

It is worth again emphasizing that the framework display scheme of FIGS.3/4 can easily be modified to support transitions to a display ofmultiple sub-sets of cells 312. This may include multiple sub-sets ofcells 312 from the same level (e.g., as in the same level from thelevels 506-512 of the schematic FIG. 5). In one embodiment of thisfunctionality, selecting an instance of open space 360 (only someexamples of such space are identified in FIG. 3) causes a directtransition to a corresponding display of multiple or all sub-sets ofcells 312 on the selected level (as compared to screenshot 400 in FIG.4, which only shows one sub-set of cells 312). Again, a mechanism fortransitioning back to a previously displayed framework view is likely tobe included as a user-selectable option. This could be but is notlimited to a mechanism similar to that shown in FIG. 6 and/or amechanism made accessible by way of one of buttons 314-322.

Those skilled in the art will appreciate that application 104 isconfigured to change the size and/or presentation of sets of cells 312,the cells 312 themselves and/or depictions of framework levels (e.g.,levels 506-512) as necessary to efficiently utilize available screenshotspace in a visually appealing way. For example, the described transitionfrom a display similar to screenshot 700 to a corresponding display ofmultiple or all sub-sets of cells 312 on a selected level is, in oneembodiment, accomplished by terminating the display of the non-selectedlevels (and associated sub-sets of cells 312) and then expanding thedisplayed size of the selected level (and associated sub-sets of cells312) to more efficiently fill available display space. This is not tosay that the non-selected levels need necessarily be removed in theirentirety. It is within the scope of the present invention for thenon-selected levels (and associated sub-sets of cells 312) to bede-emphasized without being removed (e.g., they can be grayed out, beotherwise made inactive, shrunk, moved to a secondary position, etc.).When an indication of the non-selected levels does remain on thedisplay, the selected level may at least be given a more prominentposition and/or more prominent display characteristics.

In one embodiment, the display characteristics of a selected level areautomatically adjusted so as to make its own included selectablecomponents easier to select in terms of the user dexterity required toaccomplish the selection. For example, the size of the touch, click,etc. selection controls area associated with each of the sets of cells312, the individual cells 312, and/or any selectable empty space areillustratively increased in size in order to make them more easilyactuated. The opposite is true when more framework components are addedwithin a given screen shot—sizing is reduced in order to accommodatemore information on the screen at the expense of smaller componentsbecoming more difficult to select (or, in one embodiment, evenimpossible to select without re-sizing).

Up to this point, the re-sizing and re-configuring of displayedframework 164 components and related selection controls has been mostoften described as being automatically effectuated by application 104 inresponse to user-initiated navigation transitions between differentframework views. In accordance with another aspect of the presentinvention, the system is configured to support such adjustments inresponse to reconfiguration commands received from the user. Forexample, a user is illustratively provided with the ability to use auser input mechanism (e.g., a mouse, touch input, multi-touch input,voice command input, etc.) to communicate commands that cause displayedcomponents of the framework to be rotated, re-sized and/orre-positioned. It is also within the scope of the present invention forthe system to automatically or in response to input (e.g., user input)change the perspective of a cell, group of cells, multiple groups ofcells, etc. For example, but not by limitation, the depiction of cells312 shown in FIG. 4 can be modified from the shown two dimensionaldepiction to a three dimensional representation or a perspective viewsimilar to the perspective utilized to show the various framework levelsshown in FIG. 7.

Screenshot 700 is illustratively configured such that the user can,utilizing a user input device to communicate user input commands,manipulate the depictions of any of the framework levels 506-512 alongwith their associated sets of sells 312. In one embodiment, any of thelevel depictions can individually or collectively be increased ordecreased in size. For example, this could be accomplished by way of atouch-input gesture involving a spreading apart or moving together oftwo distinct touches (i.e., moving fingers apart or together in a mannersimilar to that currently used in the art to re-size photographs, etc.).When the level depiction is re-sized the associated cells 312 and/orrelated selection input control mechanisms are also re-sized, as waspreviously described. Further, any of the level depictions canindividually or collectively be rotated (e.g., tipped forward, backward,or side-to-side) in terms of the perspective of their 3D appearance.Still further, the level depictions can be re-arranged, for example,through utilization of a “select-and-drag” functionality, as is known inthe art. In one embodiment, the user interfaces include a mechanism thatenables the user to return components they have moved and/or resizedback to a default configuration for the currently displayed componentsof the framework.

In another embodiment, the system is configured to make adjustmentsautomatically in response to reconfiguration commands directed by theuser at a particular cell 312 or collection of cells 312. Referring toFIG. 3, the display is illustratively configured to allocate an amountof space given to one or more of the displayed cells 312 in response toa user-initiated input causing a re-sizing of one or more cells 312. Forexample, space on side 308 is illustratively increased (and likely thesize of cells 312 and associated input control mechanism) when the spaceon side 312 is decreased. In other example, a single cell 312 that isre-sized by a user causes other cells 312 to resize (both their actualsize and the size of their associated input control mechanism).Referring to FIG. 7, a user-initiated reconfiguration of one of thecollections of cells 312 illustratively triggers an automaticaccommodating reconfiguration of other displayed framework components.This system managed reconfiguration can be for the purpose of maximizingthe efficiency with which display space is used, to automaticallymaintain a logical display of information, or any other controllingprinciple as desired by an involved system developer and/oradministrator.

Up to this point, discussion has for the most part been focused uponbroad characteristics of the VisualY model framework 164 and user-drivennavigation through various components of it. What has not been discussedis any particular ontological scheme for organizing the components ofthe framework. In that regard, it is certainly true that the scope ofthe present invention is not limited to any one particular scheme.However, at least for the purpose of being complete in terms of thedepth of the present description, examples of particular schemes willnow be provided.

In one embodiment, each different cell 312 (loosely represented in FIG.5 by corresponding model segments 502) represents a different segment orcategory of subject matter. An identifier or identifiers (e.g., but notnecessarily a symbol or symbols, a color, a sound, an animation, etc.)may be included within a cell 312 as an indication of the nature of thesubject matter relevant to the cell and/or the nature of the subjectmatter relevant to other cells 312 navigationally connected to the cell,such as but not limited to a collection of cells 312 in the nextadjacent hierarchical level of the framework. In accordance with oneaspect of the present invention, the subject matter distribution acrossall cells 312 included in the model framework 164 is consistent with anoverall objective of organizing and presenting enough subject mattercells 312 to intuitively present a relatively comprehensive and wellorganized presentation of a broad range of various forms of existence.This is not to say that all manifestations of a model framework 164 musthave this objective. Other framework manifestations could be narrower oreven broader in scope. But at least for the purpose of providing aparticularly useful example, the broad range of forms of existence willbe assumed. This model framework scheme is particularly well suited forthe dynamic, interactive, visual and auditory nature of VisualYrepresentations included in instantiations of the VisualY language. Thescheme is especially suitable for such purposes because it providesabundant opportunities for showing connections and relationships betweenvarious forms of existence.

A good place to begin a description of the exemplary “forms ofexistence” model framework is to take a more detailed look at the levelof the model framework 164 shown in FIG. 3. The exemplary frameworklevel shown in this screen shot is illustratively the main screen of ahuman centric view of the “forms of existence” framework scheme. Thecircled “Y” area 326 is illustratively the human focus of the modelframework. As will be described below in greater detail, a user has theoption (e.g., accessible by way of one of buttons 314-322) to change toa view other than the human centric view (or to a different variation ofa human centric view) and thereby change the presentation andorganization of cells 312 on any or all levels of the model framework164. However, the view shown in FIG. 4 is illustratively the primaryhuman centric view.

The cells 312 on the right-hand side of line 306 in area 310illustratively pertain to fundamental matter/energy/universe categoriesof subject matter. There are six vertical rectangles, each containing anindication (in this case a symbol or symbols) of the subject matter ofthe rectangle and/or of sub-cells 312 included in an adjacent (e.g.,connected by way of a direct navigation path) but different collectionof cells 312, as was described above. Those skilled in the art willappreciate that any of the vertical rectangles are configured tosupport, as has been described herein, automatic or user initiatedre-sizing. In one embodiment, the system is configured such that a usercan communicate a re-sizing input (e.g., but not limited to, a twofinger, multi-touch pinching open or shut similar to how photographs arecurrently known to be re-sized) that causes one or more of therectangles to be increased or decreased in size. It is within the scopeof the present invention for any or all cells 312 (i.e., any of theselectable segments on either side of line 306) to be automaticallyre-sized or re-sized in response to user input.

In one embodiment, the system is configured such that the number ofindications sub-cells 312 included in a given rectangle or any of theselectable segments on either side of line 306 may change incorrespondence with its re-sizing. For example, if the rectangle oranother segment is made large enough, symbols representing cells 312from multiple cell collections on the same or different framework levelsmay pop up. The larger the rectangle, the more detail that pops into therectangle to indicate what other sub-cells or related cells areaccessible by way of navigations through the rectangle. In oneembodiment, this re-sizing functionality is incorporated into any or allof the framework areas described herein. For example, the cellcollection shown in FIG. 4 can illustratively be similarly resizedautomatically or in response to user input so as to change the level ofdetail presented in terms of what cell collections are accessiblethrough the depicted framework component. Similarly, any of the squares,rectangles, triangles and other shapes within the FIG. 3 depiction canbe similarly resized. The system is illustratively configured toautomatically reorganize a depicted framework representation so as tooptimize the use of space and to make the depiction visually appealingand logical, and to present selectable components as easy to select aspossible. In one embodiment, this means shrinking, re-representing(e.g., shown as minimized), or deleting one or more framework areas asanother framework is increased in size. Or, framework areas can beenlarged, re-represented or re-rendered as another framework area isdecreased in size.

In one embodiment, when an area is made large enough, it will eventuallysweep in not just the directly accessible adjacent framework levels andcell clusters but also levels and clusters connected beyond the directlyaccessible. This illustratively means a depiction within an enlargedframework area that distinguishes between those framework levels and/orcells that are directly accessible and those that are not. In oneembodiment, not by limitation, this depiction is a representation of ahierarchy such as a tree diagram.

In one embodiment, changes other than re-sizing are also supported. Forexample, a framework area or a collection of cells 312 canillustratively be removed or minimized. Similar to how a user of themodern Microsoft Windows operating system is able to minimize a desktopwindow in order to open up desktop space, a user of the presentlydescribed application (or an automated system function), canillustratively minimize one or more portions of the framework 164. Or,similar to how a user of Windows is able to “x out” of an application orwindow, a user of the presently described application (or an automatedsystem function), can close one or portion portions of the framework164. In one embodiment, a “return to default” function is provided tothe user as a mechanism for easily undoing adjustments the user has madeand returning all or a portion of the framework 164 to a default state.

Returning again to the description of the exemplary ontological aspectsof the “forms of existence” framework scheme, the six rectangles in area310 illustratively represent six broad categories of fundamentalmatter/energy/universe subject matter. The six categories are organizedso as to present, to some extent, a right-to-left spectrum offundamental matter/energy/universe building blocks of which most formsof existence are comprised. The spectrum is illustratively organizedsuch that items classified further to the right are generally morefoundational and small while items classified further to the left aregenerally more complex. The spectrum therefore becomes more complex asit gets closer to the human area 326, which is consistent with the factthat humans are often understood to represent one of the most complexforms of existence. Category segments on the left of line 306 in area308 are generally human-oriented while category segments on the right ofline 306 in area 310 are generally not human-oriented except to theextent that they are building blocks of human-oriented concepts.

In one embodiment, the six categories of subject matter (eachrepresented by a separate one of the rectangles in area 310) included inthe spectrum, along with exemplary sub-categories (e.g., thesub-categories reflected in associated navigationally accessiblehierarchical constructs) are:

-   -   1) Primary building blocks (rectangle furthest to right)        -   sub-cells include: fundamental building block components            such as but not limited to light, gravity, sub-atoms,            energies, forces, etc.    -   2) Vertical periodic table (rectangle 2nd from the right)        -   sub-cells include: groupings of elements based upon any            factor such as but not necessarily limited to function,            reactivity, weight, toxicity, value, etc.    -   3) Non-generative items formed of periodic table components        (rectangle 3rd from right)        -   non-generative in the sense that there is generally no            contribution to life forms        -   sub-cells include: groupings based on any factor such as but            not necessarily limited to compounds, mixtures, etc.        -   illustratively sorted from small items up to large items (as            large as planets, cosmos, etc. or even larger)    -   4) Generative things made out of periodic table components        (rectangle 4^(th) from left)        -   generative in the sense that there is contribution to life            forms        -   illustratively sorted from small items up to large items    -   5) Microorganisms that utilize or comprise the generative things        (rectangle 5^(th) from left)        -   sub-cells illustratively organized based on size, reactivity            (inert or highly reactive thing), etc.        -   organisms in this category generally have no brain        -   includes access to cells related to environments, etc.    -   6) Macro organisms that have brain (rectangle 6^(th) from left)        -   sub-cells include illustratively organized based on size,            capacity, functionality, etc. (humans highest)        -   includes access to cells related to environments, habitats,            etc.            -   illustratively includes access to cells related to human                built things (cars, compounds, bridges, etc.)

At this point, it is worth noting that mutual exclusivity is notrequired. For example, a single sub-cell or symbol can be assignedwithin multiple categories (e.g., can be assigned within more than one“parent” cell 312). If a sub-cell or category seems to fit withinmultiple of the rectangles in area 310 then it is acceptable for it toappear within multiple of the rectangles. Thus, it stands to reason thata single navigational path can be made accessible from multiple of therectangles. Further, these principles of non-exclusivity extend to allcomponents of the model framework 164. A component of one of a rectanglein area 310 could also appear in a cell 312 within area 308 or viceversa if that makes sense in terms of the ontological logic of theframework scheme or, in particular, if that makes sense in terms of theontological nature of things that exist in the universe. In oneembodiment, the goal is to remain as consistent as possible with theactual organization of the universe, which is a complex endeavor thatrequires the noted flexible approach.

Before proceeding, it is worth digressing for a moment on the role thatreality plays within a VisualY representation, the VisualY modelframework, or other components or instantiations of the VisualYlanguage. Within application 104, reality is illustratively anadjustable concept. It was stated in the previous paragraph that, in oneembodiment, the goal is to remain consistent with the actualorganization of the universe. However, in one embodiment, reality is aparameter that may be adjusted programmatically within the software tosome extent. In other words, the system provides a function (e.g, madeaccessible by way of one of buttons 314-322 or otherwise) that enablesthe user to selectively define variables in relation to a set of rulesdefined within the software, the set of rules effectively becoming aparticular reality that is imposed programmatically on the configurationof the VisualY model framework and/or as appropriate to other VisualYlanguage components. The default state of the system is illustrativelythe current human understanding of the universe and everything in it,and thus the default goal is to conform the VisualY language componentsto that interpretation of reality. However, in one embodiment, the usercan adjust to a different reality which may be even be an unrealistic ornon-existent reality. This may be as simple as adjusting a rule ofphysics (e.g., eliminating gravity from the universe) or as fantasticalas setting the reality to a fictitious place such as a literary locationlike the Kingdom of Narnia. In one embodiment, when the user manipulatesor otherwise leaves the default reality, the user interfaces ofapplication 104 are modified to include a clear indicator of the realityin which the system is set. In one embodiment, a mechanism such as abutton is provided in order for the user to quickly return the system toits default reality state. In one embodiment, mechanisms are providedwithin application 104 to enable a user to create and save new“realities” for later retrieval. For the purposes of the presentinvention, to the extent that the term “reality” is used in the presentdescription, the term is intended to mean the applicable reality imposedat any given time within application 104.

To the left-hand side of line 306 in area 308, the cells 312 in area 326are organized around what is herein referred to as a “yperson”construct, signified by the three lines extending from a central circleso as to form a “Y” shape. As has been discussed the view of the “formsof existence” framework reflected in FIG. 3 is human-focused. This viewis especially well suited for human cognition at least because humansinherently tend toward a human-centric perspective. However, the modelis designed to effectively support VisualY representations of how, givena particular context of interest as selected by the user, the generallyhuman-centric subject matter cells 312 in area 308 directly relate toand are connected to the categories of fundamentalmatter/energy/universe subject matter cells 312. This is illustrativelyan instantiation of the VisualY language. Thus, the VisualY languageenables a “big picture” dynamic representation of a complex system, therepresentation being intuitive and capable of showing simultaneousinterplay of multiple or even numerous variables at work within thesystem.

The “Y” in area 326 illustratively divides the “yperson” into multiplecells 312 representing multiple dimensions of the yperson. The cell 312on the bottom right identified with a 328 designation is illustrativelythe corporeal dimension. Within FIG. 3, no symbols or sub-celldesignations are shown in area 328 but it is assumed that through thisparticular cell 312 the user can see and access sub-cells representingcategories of subject matter consistent with corporeal characteristicsof the yperson. For example, the cell identified as 328 canillustratively be expanded (e.g., using a multi-touch “pinch gesture) ornavigated to (e.g., navigating to a next level similar to a FIG. 4 nextlevel) in order to see more detail about related sub-cells.

The cell 312 on the top of the “Y” identified with a 330 designation isthe consciousness (awareness) dimension. Within FIG. 3, no symbols orsub-cell designations are shown in area 330 but it is assumed thatthrough this particular cell 312 the user can see and access sub-cellsrepresenting categories of subject matter consistent withconsciousness-oriented characteristics of the yperson. For example, thecell identified as 330 can illustratively be expanded (e.g., using amulti-touch “pinch gesture) or navigated to (e.g., navigating to a nextlevel similar to a FIG. 4 next level) in order to see more detail aboutrelated sub-cells.

The cell 312 on the bottom left of the “Y” identified with a 332designation is the sub-consciousness-oriented dimension. This dimensionillustratively encompasses inherent motivations that drive humans. Forexample, a human may not know what they are about to say until they sayit. It is a sub-conscious mechanism that affects what is said. This isbut one example. Within FIG. 3, no symbols or sub-cell designations areshown in area 332 but it is assumed that through this particular cell312 the user can see and access sub-cells representing categories ofsubject matter consistent with sub-consciousness-orientedcharacteristics of the yperson. For example, the cell identified as 332can illustratively be expanded (e.g., using a multi-touch “pinchgesture) or navigated to (e.g., navigating to a next level similar to aFIG. 4 next level) in order to see more detail about related sub-cells.

The cells 312 in the small corners 362 around the “Y” illustrativelyrepresent the yperson's applicable environments. Within FIG. 3, nosymbols or sub-cell designations are shown in corners 362 but it isassumed that through these particular cells 312 the user can see andaccess sub-cells representing categories of subject matter consistentwith environmental considerations relative to the yperson. For example,the corner 362 cells can illustratively be expanded (e.g., using amulti-touch “pinch gesture) or navigated to (e.g., navigating to a nextlevel similar to a FIG. 4 next level) in order to see more detail aboutrelated sub-cells. In one embodiment, each of the corners 362 (andassociated sub-cells 312, etc) represents a different “level” ofenvironmental considerations, illustratively on a spectrum from micro tomacro (e.g., from the micro environment to macro environment).

Above the yperson area 326 are a plurality of horizontal rectangles in anon-tangible human construct area 340. Each rectangle in area 340 isillustratively a different cell 312, although only a representative fewhave been labeled in the FIG. 3. The cells 312 in area 340illustratively represent, within the model framework, non-tangible“things” created by humans. For example, it is through a cell 312 thatthe user might access a cell 312 representing the culture of aparticular city (e.g., but not necessarily depending upon user-selectedapplication preferences, the culture of the city where the currentlyselected ypserson(s) live). Other cells also lead to cells 312 thatrepresent how people think (e.g., but not necessarily depending uponuser-selected application preferences where the currently selectedyperson(s) lives). Other examples include what is believed, how theyperson(s) have organized (institutions, etc.), normals, rules, laws,governments, etc. Of course, these are only examples and the scope ofthe present invention is certainly not so limited.

Notably, the rectangles in area 340, which generally representnon-tangible things, are horizontally oriented. In contrast, therectangles in area 310, which generally represent tangible things, arevertically oriented. In one embodiment, this theme is carried throughoutthe application interfaces. In one embodiment, a rectangle oriented atan angle represents something where the tangible/non-tangible line isambiguous or not clear.

To the left of the yperson area 326 are a plurality of circles in apeople area 364. Each circle in area 364 is illustratively (though notnecessarily) a different cell 312, although only a representative fewhave been labeled in the FIG. 3. The cells 312 in area 364illustratively represent, within the model framework, people having arelationship to the yperson, or at least having interaction on somelevel with the yperson. Each circle illustratively represents one personor a group of persons. The representation may be very specific (e.g., acircle representing single specific person such as President BillClinton) or may be collective or otherwise generic (e.g., a circlerepresenting farmers from Kansas). Some cell 312 circles included in oraccessed through area 364 (e.g., sub-cells 312 made accessible by way ofnavigation through area 364 cells) will represent individual humans(e.g., Paul Johnson from Toledo, or a postman, etc.) and others willrepresent groupings of humans (Boy Scout troop 345 from Toledo, orcommunists, etc.).

Those skilled in the art will appreciate that just as the system isconfigured to enable the user to navigate through the rectangles in area310 (or through any other cell 312 clusters in the model framework), sotoo can the user navigate through the circles in area 364, for exampleto other circles or even to portions of the model framework outside ofarea 364 if logically appropriate. It is therefore to be understood thatall of embodiments described herein at least in terms of the use ofsymbols, navigation methods and the like described in the context ofother cells and cell clusters in the model framework are just asapplicable to the circles in area 364. Still further, just as verticalrectangles and horizontal rectangles used within the model frameworksignal relevance to a particular area within the ontology of the modelframework, so too does the use of circles illustratively signal aconnection to a people-oriented relationship to the yperson.

It is within the scope of the present invention for a circle or circleswithin the model framework to represent living people or dead people.Circles may represent people that exist or people that do not exist ornever have existed (e.g., fictional people). They may represent peoplefrom the past, present or future. Further, the term “people” should beliberally construed. Circles may also represent cartoon characters(e.g., Scooby Doo), animals (e.g., a favorite pet), or any other person,place or thing that has the capacity to function in a manner similar toa person in terms of their interaction with, or relationship to, theyperson. By not strictly requiring a circle to represent a living human,the system is flexible enough to accurately reflect the reality (thereader should be reminded that, as state above, reality is aprogrammatically adjustable concept within application 104) of even anunusual set of circumstances. Further, as has been described, the systemis also flexible enough to allow an entity represented by a circle toalso be represented elsewhere in the model framework. The goal is, inone embodiment, to configure the ontology of the model framework so asto accurately reflect reality (the reader should be reminded that, asstate above, reality is a programmatically adjustable concept withinapplication 104). Thus it is undesirable to impose rules that mightsimplify the model framework at the expense of departing from reality.

In one embodiment, the configuration and organization of the cells 312in area 364 may be but illustratively is not random or arbitrary.Instead, the configuration and organization of the circles in terms oftheir placement relative to each other, their placement relative to thetop or sides of area 364, their placement relative to yperson area 326,their motion, their color, their associated sounds, their size, or anyother adjustable parameter is indicative of a characteristic of arelationship or level of interactions with the yperson. For example, inone embodiment, the directness of an interaction is factored into thescheme utilized to organize the cell 312 circles. Humans with the mostdirect interaction are illustratively touching or at least closest toyperson area 326.

In another embodiment, the depiction scheme of circles within the modelframework includes a mechanism for indicating whether an exchange withthe yperson is direct or indirect. For example, an exchange mediatedthrough another entity is illustratively accounted for in the depictionscheme. In one embodiment, assuming such a scenario is consistent withreality (the reader should be reminded that, as state above, reality isa programmatically adjustable concept within application 104), amediated exchange may even be made shown in relation to a circlerepresenting someone no longer living. For example, if a now dead personthat never met the yperson made music years ago that is listened to bythe yperson that can be considered a mediated exchange and isillustratively shown as such within the model framework. In oneembodiment, mediated exchanges are indicated by utilization of aslightly different shape (e.g., an oval instead of a circle), color,sound, placement, etc. Further, less direct exchanges (such as but notlimited to mediated exchanges) are illustratively assigned relativelysmaller cells 312 and/or are positioned further away from the ypersonwithin the model framework. Of course, these are only examples and thescope of the present invention is certainly not so limited.

Below the Y area 326 is an area 366 dedicated to depicting an applicablespatial action or otherwise a context for an active VisualYrepresentation. In other words, area 366 provides a way to show anaccurate context over space and/or time for the currently active VisualYrepresentation. For example, a video of a person jumping out of a planemay be shown in area 366 while, as part of an applicable instantiationof the VisualY language, many cells 312 throughout the VisualY frameworklight up, change color, move, generate sound, become associated with ananimation or symbol, and otherwise show, on the potentially manyrelevant levels of the framework, what is happening to the yperson whilethey are jumping. Of course, this is but one of an unlimited number ofexamples of how area 366 is utilized to indicate a context for activity(e.g., sounds, coloration, changes in sound, changes in color,animations, etc) occurring in relation to cells 312 in the framework164.

In one embodiment, the context provided in area 366 is active so as todepict a dynamic context for changes occurring relative to the frameworkas part of a VisualY representation. Thus, the provided context can be avideo, or an animation, or a series of photos, or an arrow or otherpointer moving across any of a variety of different types of timelines(e.g., a human development timeline, a date-oriented timeline, or atimeline organized around any other concept). Alternatively, the contextprovided in area 366 can be static such as a picture or a map. Thecontext provided in area 366 may be made in terms of the places orevents in the past, present or future. It can be a depiction of eventsthat did already occur, events that have never occurred, events thatwill but have not yet occurred, events that will never occur, orotherwise. The context may be presented in terms of real places orplaces that do not exist in the “real” world, or even fictitious places.Or, it can be an event such as World War II or Super Bowl VII. It iswithin the scope of the present invention for the context provided inarea 366 to be absolutely any mechanism for depicting a space, time,subject matter or any other context for what is being represented inrelative to the framework 164 as part of an active instantiation of theVisualY language (i.e., an active VisualY representation).

In accordance with another aspect of the present invention, just likeother cell 312 areas within the framework are configured to supportnavigation to other levels of detail by navigating to other cell 312areas, so too may area 366 be configured to support navigation to otherlevels of detail. In one embodiment, the system is configured to enablea user to choose from a variety of different perspectives of the samecontext for a VisualY representation. For example, the user mightnavigate from a pulled back video view of a person sitting on a bencheating corn to a close-up video view of the person's jaw chewing corn.In one embodiment, the system illustratively includes mechanisms forslowing down (e.g., slow motion), speeding up, re-starting, rewinding,pausing, or otherwise manipulating a video or animated area 366 context,thereby illustratively causing a corresponding (e.g., slowing down,speeding, pausing, rewinding, etc.) change to the current dynamicVisualY representation. These are but examples of how the area 366 areacontext can be selectively adjusted without actually changing the activeVisualY representation.

However, it is also within the scope of the present invention for thesystem to be configured to utilize navigation of area 366 (or similarfunctionality accessed elsewhere within the system such as throughfunctions associated with one of buttons 314-322) as a mechanism thatenables the user to switch from one context for an active VisualYrepresentation to a different context for the VisualY representation,thereby effectively transitioning from one VisualY representation toanother. For example, the context area 366 might be changed from oneperson eating corn in a “right now” sense to a specific infant childeating corn for every meal for a month. This would not only change thecontext for the currently active VisualY representation but would alsochange the VisualY representation itself (e.g., it would change theyperson area 326 from the one person yperson focus to the infant childfocus, along with the many other likely changes to the VisualYrepresentation in relation to model 164 that that would produce).

When the context area 366 is navigable and/or alterable, the systemillustratively provides an indication such as a symbol, color or soundto indicate navigability. In one embodiment, area 366 is navigable andotherwise can be manipulated in at least all of the same ways describedherein in relation to cells 312 included within the framework 164. Forexample, different context options might be accessible through area 366in the form of cells 312 made accessible through area 366. In oneembodiment, one or more cells (even potentially multiple cellssimultaneously) accessed through area 366 can be selected so as to setthe active context for the currently active VisualY representation.

In one embodiment, any or all of the cells 312 in framework 164 can bemade selectable such that, when selected by the user, correspondingchanges are instantly made to the currently active VisualYrepresentation. Thus, the user is illustratively able to navigatethroughout the framework selecting and/or deselecting cells 312 asdesired and thereby configuring the currently active VisualYrepresentation as desired. In one embodiment, the user can start anactive space/time video context in area 366 at the beginning of a videothat is the context for a first VisualY representation that is madewhile the video is playing. Then, the user can pause the video whilenavigating through model 164 and making changes by selecting andde-selecting cells 312. Then, the user can manipulate a control providedwithin the system in order to re-start the video. An updated VisualYrepresentation will then be shown in relation to the re-started video,the representation being updated due to the new set of variables appliedto the representation by way of the new combination of selected andde-selected cells 312. In other words, the behavior of the VisualYrepresentation (e.g., colors, sounds, animations, motions, etc. shown inrelation to the framework 164) is programmatically dependent upon systemvariables as selectively set by the user of the application. In oneembodiment, the behavior of the VisualY representation is alsoprogrammatically dependent upon which view of all or a portion of theframework 164 the user has selected.

4. The VisualY Engine

As should be becoming apparent, application 104 is configured to providea richly dynamic and programmatically responsive user experience in theform of instantiations of the VisualY language. A given instantiation islikely to include a VisualY representation of information, whichillustratively includes sound, color, multimedia, motion (or any othersimilar means for conveying information) provided in relation to any orall of the components of the VisualY framework. As the user navigatesthrough the framework, as the user selects and/or deselects componentsof the frameworks, and as the user changes views of all or portions ofthe framework, the relevant VisualY representations (i.e., color, sound,multimedia, motion, etc.) are configured to programmatically respond andbe presented to the user in a way that communicates knowledge andinformation in the context of the currently presented VisualY frameworkcomponents. It should also be noted that, in one embodiment, otherprogrammatic variables may come into play, such as other user-selectedparameters (e.g., in one embodiment, the user selects a purpose fortheir current exploration of the VisualY instatiation) that will impacthow the framework and/or representation responds programmatically.

In accordance with one aspect of the present invention, the application104 component that drives the dynamic responsiveness of instantiationsof the VisualY language is a VisualY engine 170, which is shown in FIG.2. It is a misnomer of sorts to refer to engine 170 as a singlecomponent because, in implementation, it is likely comprised of avariety of different hardware and software sub-components.

Engine 170 includes, in one embodiment, front end software componentsthat drive the front end of application 104. For example, the front endcomponents determine the user experience in terms of the sounds, images,multi-media, content, functions, controls and tools presented to a givenuser automatically, semi-automatically or in response to user input orotherwise. In one embodiment, the front end components include primarilyJavaScript coded components that communicate with backend systemcomponents responsible for actually retrieving information from thedatabase(s) as necessary to appropriately fulfill a frontendrequirement.

In one embodiment, the front end software components of engine 170include WebGL programmed components that extend the capability of theJavaScript programming language so as to enable engine 170 to generateinteractive application 104 graphics (e.g., interactive 3D graphics)within the browser of users of the application. This WebGL codeillustratively executes on the computer display cards of the users.Those skilled in the art will appreciate that WebGL is a context of thecanvas HTML element and, accordingly, components of the front endsoftware components of engine 170 are coded in accordance with canvasprogramming constructs.

In one embodiment, the front end software components include audio APIcomponents that extend the capability of the JavaScript programminglanguage so as to enable engine 170 to dynamically generate interactiveapplication 104 audio in logical relation to visual informationpresented through the user's browser. The audio API components enablemanipulations (changes in pitch, tempo, etc.) of audio signals on thesound cards of the users. Accordingly, in one embodiment, the audio issynthesized and processed directly in JavaScript without use of aplug-in. In one embodiment, audio is managed through the AudioData API(Mozila Firefix) and/or the WebAudio API (Chrome and Sarari) therebyenabling audio information to be manipulated more directly or at leastrelatively more directly upon the user's hardware responsible for audiodata management.

In one embodiment, certain of the front end software components,including the components for managing audio and video features presentedto an application user, are coded so as to be conditional in the senseof being rules based but are not so conditional as to require codedanticipation of every possible outcome. The described JavaScript API'sare illustratively applied within the engine 170 framework so as toenable a user of application 104 to manipulate variables (e.g.,manipulate manifestations of the VisualY language by selectingde-selecting, or navigating through sets of cells 312) that affect inessentially real time what is presented to them in terms of their audioand visual application experience. The JavaScript API's enable this tobe accomplished without working through a browser plug-in such as aFlash plug-in. By enabling the audio and visual experience to beefficiently managed and manipulated by the user's own audio and videohardware, the JavaScript API's enable the front end components toprovide what is essentially a generative user experience. In oneembodiment, the JavaScript API's are utilized within the front endsoftware components so as to provide a visual and/or an audio userexperience powered by some degree of programmatic artificialintelligence (or at least an allusion from the user's perspective ofartificial intelligence) instead of a strictly rules based logicalapproach. Thus, not every possible outcome (e.g., not every possiblecombination of activated or deactivated cells 312) need beprogrammatically accounted for per se in order for there to be a logical(at least from the user's perspective) manipulation of the userexperience.

It should be said that these generative components of driver 170 maystill technically be conditional in terms of their logic but they alsosometimes will programmatically accommodate complexity or chaos to somedegree. In one embodiment, the algorithms utilized to drive theexperience of a user of application 104 facilitate production of userexperiential outcomes that were not strictly programmaticallyanticipated when the code was originally developed. In anotherembodiment, the system is programmed to have an ability to “learn” thatis implemented through a variety of artificial intelligence algorithmsand techniques. Then, an outcome that is learned may illustratively bere-created differently under a same or similar set of conditionssubsequently established by the user through manipulations made duringinteraction with the application 104.

To summarize, the experience of the user of application 104 asfacilitated by engine 170 is interesting in terms of how it is deliveredthrough the user's web browser on a user-driven basis (i.e., responsiveto user adjustment of application variables) so as to be dynamicallygenerative and responsive thanks at least in part to direct generationand/or manipulation of audio and/or visual data upon (or at least in aclose direct relationship with) the user's audio and video hardware.This is illustratively accomplished without reliance upon functionalitydelivered by way of a web browser plug-in (e.g., a Flash plug-in) orsome other similarly delivered additional functionality. While the scopeof the present invention does not exclude a plug-in drivenimplementation such as a Flash-based implementation, such a solution isassumed to be less desirable, for example, because the graphicsthroughput is considerably less and there will generally be limited orno graphics shading options. The options for presenting representationsof information-rich, dynamic complex systems, and for enabling multiplepeople in remote locations to collaborate in real time in the continuousdevelopment of such representations, are far greater and more flexibleutilizing the non-plug-in implementations described herein than theplug-in driven solutions that, at the time of this writing, currentlyare the dominate choice by far for navigating the more dynamic and“alive” sites on the world wide web and on other similar informationalnetworks.

It is worth elaborating some on the collaboration aspect of embodimentsof the present invention. The experience of the user of application 104as facilitated by engine 170 illustratively revolves around a communitythat collaborates essentially in real time. Engine 170 is configured toenable the users of application 104 to collectively and remotely create,organize, collect, modify, transform, present, navigate, browse, search,experience, display, store, retrieve, transfer, and otherwise interactwith instantiations of the VisualY language. In one embodiment, theVisualY representations 162 (e.g., presented to a user in the context ofall or a portion of the VisualY model framework 164) are made availableon the World Wide Web such that the described and other functions ofengine 170 are made publically available to some extent.

Accordingly, embodiments of the present invention pertain toapplications of relatively new browser technology that enable relativelylarge graphics (and audio) throughput. The system enables a connectionof easy to use and flexible programming code to an especially efficientpipeline to a user's hardware where it is quickly executed. This directpipeline scheme enables much more throughput than a traditional approachwhere code is first move around and processed through higher levelswithin the computing architecture. For example, the direct pipelineapproach enables the skipping of more traditional steps such as settingup up binary and library resource requirements, compiling to binary,fixing cross operating system parameters, etc.

The described browser-driven but not plug-in driven programmaticapproaches (such as WebGL) are essentially pathways to getting codedirectly onto hardware, thereby making it much faster to execute. Thistechnology is directly responsible, in accordance with one aspect of thepresent invention, for empowering the user community to directly orindirectly participate in a collaborative effort to manipulate the code(e.g., manipulate instantiations of the VisualY language) and saveprogress as such manipulations are made. The users thereby gain fromeach other as value (i.e., in the form of information and knowledgecommunicated in accordance with the constructs of the VisualY language)is added and modified on a collaborative basis. And this is still doneutilizing a browser driven remote application approach rather thananother approach, for example, requiring each user to obtain a standalong application.

In one embodiment, the VisualY language includes a code based frameworkwith styling principles, macro functionality for task accomplishment,web failbacks for cross browser support, etch. The engine 170illustratively utilizes these and other components of the VisualYlibrary and framework. The engine is an instantiation of code thatillustratively comes from the VisualY library and framework. Thus, itcan be difficult to draw exactly the line between the language and theengine. However, exactly where this line lies is not necessarilycritical to the present invention.

FIG. 8 is a block diagram showing, in accordance with illustrativelyembodiments of the present invention, components of engine 170. Engine170 is illustratively an instantiation of VisualY core components 802.In essence, engine 170 is the driver and has its hands in (and reliesupon) the functionality of the core components 802. In one embodiment,core components 802 provide the visual language manifested as JavaScript(though the scope of the present invention is not so limited). Thesecomponents are illustratively the API's of the application 104 systemand therefore define the basic application components.

Included under the core components 802 are DOM (document object model)utilities 804. This is illustratively where information is provided formanipulating the page structure within application 104. These componentsare illustratively coded as CSS3 and/or HTML5 constructs, though thescope of the present invention is not so limited. Also included are datautilities 806, where functionality is provided for manipulating datathat is to be presented within application 104.

Also included under the core components 802 are charting components 808.The charting components illustratively include, but not by limitation,weighted stacks, bar charts, model handling functions, etc. This may ormay not be where rendering is handled but is illustratively an areawhere instructions are recorded to define how things are drawn. Aseparate sketching package 810 provides functions that make it possibleto draw on a very efficient basis, as has been described in relation tothe incorporation of WebGL and Canvas mechanisms into the applicationarchitecture. A set of styling components 812 define available colors,line thicknesses, etc. Those skilled in the art will appreciate that thedescription of components of engine 170 described herein is notexhaustive and that other components are likely to be included at leastto supply basic functions that are known in the art to be part of thekind of network driven application architecture described herein. Forthis reason, a catch-all box 814 is included in the FIG. 8 depiction ofengine 170.

5. First Principles Engines

In relation to the VisualY model 164 it was explained that cells 312within the model framework need not be exclusive in order to enable anaccurate, scientifically founded presentation of information andknowledge. This is almost certainly true of the cells 312 in area 326where interplay between the different dimensions of the yperson arelikely to be common. For example, the line between conscious processesand subconscious process is not necessarily going to be clear. In orderto maintain the integrity of the ontological classification scheme, itwill sometimes make sense for there to be some level of redundancy interms of the components of one cell 312 relative to another.

As much as possible, this interplay and overlap within the modelframework 164 is preferably managed not on an arbitrary or opinionatedbut instead is managed so as to remain as consistent as possible withthe interplay and overlap of corresponding processes that exist inreality. In other words, as the model framework grows and evolves, it isillustratively the goal to stay as consistent as possible with the wayin which humans, nature and other manifestations of existence areactually programmed. For example, if a human wants to control something,then an awareness of the thing to be controlled must first be created.Humans generally have some degree of control in terms of their view ofwhat is really happening within their surroundings. So, the line betweenthe conscious and sub-conscious is not necessarily always clear. By notrequiring a given cell 312 construct to be assigned exclusively to asingle location within the model framework 164, there is allowance forenough flexibility to address mysteries or ambiguities or complexitiesin terms of the underlying programming and processes that drive reality(the reader should be reminded that, as state above, reality is aprogrammatically adjustable concept within application 104).

The goal of reflecting and representing real underlying fundamentalprogramming and processes is not exclusive to the human context of themodel framework. As is likely becoming apparent, the VisualY modelframework is intended to encompass all levels of existence, from thesmallest building blocks to the largest most complex constructs. Inreality (in particular the default reality of the universe discussedabove in the context of programmatically adjusting reality withinapplication 104), every manifestation of existence is subject to somedegree of inherent fundamental programming that dictates, to someextent, interactions among and between the constructs represented withinthe model framework. By not requiring a given cell 312 construct to beassigned exclusively to a single location within the model framework,there is allowance for enough flexibility to address mysteries orambiguities or complexities in terms of the underlying programming andprocesses that drive reality (though the reader should be reminded that,as state above, reality is a programmatically adjustable concept withinapplication 104).

In accordance with one aspect of the present invention, first principlesengines 816, which are shown in FIG. 8 as part of the larger engine 170,function within the application 104 framework as the mechanism forenforcing restrictions and requirements relative to the VisualY modelframework and related VisualY representations (i.e., instantiations ofthe VisualY language) based on scientific, nature, and/or otherfundamental principles. For example, the principles upon which therestrictions and requirements are based are illustratively firstprinciples. A first principle is, in a philosophical sense, a basicfoundational proposition or assumption that cannot be deduced from anyother proposition or assumption. To the extent that new first principlesare discovered or existing first principles are abandoned or changed,corresponding changes can be made (for example, by an administrator ofapplication 104) to the engines 816 in order to keep, as much aspossible, a given VisualY instantiation consistent with a fundamentallyaccurate portrayal of all levels of existence.

In one embodiment, engine 816 will, on the basis of maintainingalignment with a first principles or other principle accounted for inengine 816, require or refuse a cell 312, a link between cells 312, acolor designation, an animation, etc. Thus, engines 816 are utilizedwithin application 104 to enforce alignment with a set of basesprinciples by policing, to some extent, interactions among and betweenthe constructs represented within the VisualY model framework.

In reality, everything (i.e., all things at all levels of existence) isin a constant state of change. Consistent with this premise,instantiations of the VisualY language provides a visual (and sometimesaudial) representation of all constructs of existence and the VisualYprogramming and language provides the tools for showing the constantchange and interactions among the constructs of existence. And afundamental assumption underlying this approach is that everything inexistence is in a constant state of change and everything in existenceis, to some extent, subject to some degree of fundamental programmingthat drives the change and therefore directly influences interactions atall levels of existence.

Thus, when the mechanisms of VisualY are applied within a given contextto model framework 164, it becomes possible to explicitly depict humanprogramming and interactions among the various dimensions of the yperson(i.e., the “person” reflected in area 326). In one embodiment, theapplication includes a function (e.g., a function accessed by way of oneof buttons 314-322, or by navigating through the constructs 312 of themodel framework and selecting a cell 312 or setting parameters inrelation to a cell 312) that enables a user to select a time independentor time dependent context to be imposed upon the VisualY representationsof interactions and programming displayed in relation to the modelframework. For example, the user can set the display to show what ishappening now, what happened during a point in history (a specific dateor a range of time, etc.), or what will happen in the future. However,selecting a time context is not limited to units of time as measured inrelation to a clock or calendar. The user can illustratively choose anymeasure of time such as an historical event (e.g., during World War II),a human event (e.g., during birth, during the yperson's second decade oflife, after the yperson eats, after the yperson plays football for 10years, etc.), or even a theoretical event that is not consistent withreality (e.g., after the yperson eats corn, or after the yperson gets aconcussion, etc.).

As is apparent in some of these examples, the application isillustratively configured to enable a user to select and/or set any ofwhat is likely to be an abundant number of parameters that have thepotential to impact the VisualY representations of interactions andprogramming displayed in relation to the model framework. While theseparameters may be set by way of functions accessed by way of one ofbuttons 314-322, it is more likely that such parameters will be set bynavigating through the constructs 312 of the model framework and simplyselecting a cell 312 or setting parameters in relation to a cell 312.The functions that dictate how different parameters will affect themodel framework and/or the VisualY representations (e.g., howinteractions between model framework constructs 312 are shown using themechanisms of the VisualY programming and language) is illustrativelybuilt into the computer code underlying the framework and/or themechanisms of the VisualY programming and language.

Accordingly, the model framework 164 and the mechanisms of the VisualYprogramming and language combine to make it possible, for example, todepict what gets into the memory of the yperson (i.e., knowledge). Andthis can be seen at different levels and with connections to otherdimensions of the yperson and with connections to other constructsoutside of the human realm (e.g., with connections to constructsincluded in the subject matter categories of the cells 312 on side 310of line 306). Further, it is also possible to depict, for example,specific types of trauma within appropriate cell 312 dimensions of theyperson and with explicit and appropriate connections to other cells 312on any level of the model framework 164.

In one embodiment, the system enables the user to set a particularyperson context for area 326. This may be accomplished throughinteraction with a function accessed by way of buttons 314-322. Also oralternatively, setting the yperson context may be accomplished bynavigating through the constructs 312 of the model framework andselecting a cell 312 or setting parameters in relation to a cell 312.The yperson context can illustratively be set to represent a specificperson or a generic human being or a group of human beings (e.g., acompany of individuals, an organization of individuals, etc.). When theypserson context is changed, the rest of the model framework 164 and itsrelated VisualY programming illustratively updates automatically inorder to modify the representation of complexity so as to be consistentwith the selected yperson focus.

In one embodiment, which is visually depicted in FIG. 3 by theoverlapping circle 334, the yperson focus of the representation can be agroup of persons each having their own yperson representation. Thesystem is illustratively configured to enable the user to selectivelyadd or remove ypersons to the “stack” of “Y” people in area 326. Asusers are added or removed, the rest of the model framework 164 and itsrelated VisualY programming will illustratively update automatically inorder to modify the representation of complexity so as to be consistentwith the selected yperson focus. Thus, depending upon the selectedsettings in area 326, the system is design to account for and representinformation in relation to collective (i.e., across a group) humandimensions (e.g., collective consciousness, sub-consciousness, etc.).Similar to the described functionality described above in relation to asingle yperson context, the mechanisms of the VisualY programming thentrigger representations of connections between the human-oriented cells312 on side 308 of line 306 and the other more foundational cells 312 onside 310 of line 306.

In one embodiment, the yperson focus in area 326 indicates a group ofdifferent individuals the application can be set to only conform therepresentation to one of the individuals at a time. For example, theuser can illustratively see layers if individuals represented in therings within area 326 (e.g., the top-most yperson representation isillustratively rendered semi-transparent so other y-persons underneathare apparent, or the edges of other y-persons might appear so as torepresent a stack of ypersons). The system is illustratively configuredto enable the user to switch between different ypersons or groups ofypersons as desired, which will illustratively initiate appropriate andcorresponding changes within the VisualY programming and languagerepresentations in relation to the model framework.

In one embodiment, the yperson focus can be set to a class of people ora very large number of people. For example, can illustratively set toall republicans in the United States, or all people in China, etc. Alsoor alternatively, it can illustratively be set to a hypothetical but notnecessarily specific person, such as an “open-minded person” or “aChristian.” In one embodiment, the yperson can be set to be the“average” person, determined by all people that the user has included ina “stack” of ypersons within are 326. In this case, the selected ypersonwould be a conglomeration of all persons included in the stack.

In one embodiment, the selected yperson can even be selected to be anon-human but living creature. For example, the yperson can be selectedto be an ant, or a breed of dog, or a specific dog. Any living thingcan, in one embodiment, be chosen as the focus of the representationwithin areas 326. However, the focus of the representation within areas326 can be something other than a living thing. It can be a deity, afictional character, an organization, a fictional organization, oranything else.

As has been hinted at, it is to be understood that some or all of cells312 support some degree of data input. For example, when the ypersoncontext is set to a specific individual, it might be implemented for apurpose such as that person's medical chart. Both the doctor and thepatient (i.e., the individual) can illustratively interact with themodel framework 164 utilizing mechanisms of the VisualY programming andlanguage so as to add information to the framework and thereby to thechart. For example, the doctor might navigate through the modelframework and add information in an appropriate cell 312 locationregarding the result of lab tests for the individual. The individualmight navigate through and add, in an appropriate cell 312 location,information as to how they are feeling and when they were feeling it, orwhat they ate and when they ate it, etc. Then, in accordance with oneaspect of the present invention, the mode framework is programmed tosome extent to respond to the entered information by automaticallyimplementing VisualY mechanisms that depict appropriate and logicalconnections to other cell 312 constructs.

Accordingly, the model framework 164 and the mechanisms of the VisualYprogramming and language combine to make it possible, for example, torepresent complexities of a human state of being (even during anydesired user-selected period of time, as described above). Further, thedirect human aspects of the state of being (i.e., shown in relation tocells 312 on side 308 of line 306) are represented in direct relation tomore fundamental aspects of the state of being (i.e., shown in relationto cells 312 on side 310 of line 306). Thus, in one embodiment, itbecomes possible to show what foods were eaten in combination with eventhe most fundamental components of those foods, and what impacts eatingthe foods have on the yperson.

It is worth emphasizing that the model framework components and therelated VisualY language and programming representations will, bydesign, vary based on commands and preferences received from the user.The user will illustratively only see the model framework level(s) andthe level of VisualY detail that they want to see, which very well mayinclude all levels. The application is illustratively configured toenable the user to hide, fold up, or re-size cells 312 as desired tocause the application to display a desired level and scope of detail.

VisualY representations when applied to the model framework 164 enableusers to see how things on all levels of nature interrelate to oneanother. Human constructs and ideas in one part of the model frameworkare actively shown (i.e., are shown in terms of their inherently dynamicand active nature) in relation to interrelated principles of thescientific foundations of the world in another part of the modelframework. This reflects the fact that everything is programmed innature, from the smallest things to the largest. Human beings have thecapacity to understand the related mechanisms that drives us and thatdrive everything. In order to tap into this understanding, one must moveaway from seeing things superficially and from a very limited point ofview. Instantiations of the VisualY language enables a new and betterperspective of how things really are in nature.

Instantiations of the VisualY language reflect the fact that forms ofexistence, including but not limited to the forms of existencerepresented within framework 164, have at their root energy that drivesatomic and subatomic existence, that drives light and gravity, thatdrives building block elements and forces, that drives what humans referto as life. Everything is in play —doing something at alltimes—activated. Everything is dynamic. Everything is complex in thateverything is, to some extent, a product of complex interaction.

Thus, instantiations of the VisualY language reflect the fact that thereis a dynamic dimension to everything. Nothing is static. Whileinstruments of science are sensitive to the constant state of change,humans on their own have a difficult time sensing it due in large partto limitations of the human senses. It should therefore probably come asno surprise that human language has not developed so as to beparticularly effective when it comes to describing the dynamic andinteractive dimensions of human surroundings. And because humans relayon language and senses almost exclusively for our understanding of theexistence around us, and within us, we have an extremely limitedday-to-day, theoretical, practical comprehension of how everything inexistence works. Current systems of language consist mainly of words anddo not easily enable one to fully and accurately imagine or describedifferent forms of existence. Current systems are inadequate at least inthat they are inherently linear and for the most part two-dimensional.

FIG. 9 is a screen shot of a multi-variable system for producing userinterface components on a generative basis. On the left hand side of thescreen shot are a plurality of user adjustable variables. Each variableis capable of being set to any of a range of different states. Changingthe state of a variable will illustratively change the motion, color,sound, and otherwise the configuration of patterns on the right side ofthe screen shot.

In one embodiment, patterns such as those on the right hand side of theFIG. 9 screen shot are examples of what parts of a VisualYrepresentation will look like. Thus, one or more of the patterns on theright hand side are depicted in conjunction with (e.g., over one or morecells) the VisualY model framework as part of an instantiation of theVisualY language. Those skilled in the art will appreciate that, whenthe patterns are incorporated into an instantiation, the changing of thevariables may be user-initiated but some or all of them are alsoautomatically adjusted programmatically. For example, the generativenature of the programming of engine 170 will illustratively drivechanges in the variables and therefore changes in the patterns andtherefore changes in the instantiation.

FIG. 10 is an example screen shot demonstrating an alternate view of aportion of the VisualY model framework. In one embodiment, the griddedarea on the right hand side of the alternate view is a substantially orentirely complete depiction of the VisualY model framework, even if thecomponents depicted on the left hand side are only a portion of themodel framework. Instantiations of the VisualY model framework areillustratively presented as VisualY representations presented inconjunctions with the model framework as shown in FIG. 10. In oneembodiment, the user is able to move different combinations ofsub-portions of the model framework in the left hand panel while at thesame time always seeing a presentation of the more or fully completemodel framework in the right hand panel. Instantiations of the VisualYlanguage are then active in that VisualY representations areresponsively and generatively presented in either or both of the panelsin conjunction with the different depictions of the VisualY modelframework.

FIG. 11 is another screen shot of a multi-variable system for producinguser interface components on a generative basis. In the small box in thetop right hand corner of the screen shot are a plurality of useradjustable variables. Each variable is capable of being set to any of arange of different states. Changing the state of a variable willillustratively change the motion, color, sound, and otherwise theconfiguration of patterns shown in the rest of the screen shot.

In one embodiment, patterns such as those shown in the bulk of FIG. 11are examples of what parts of a VisualY representation will look like.Thus, one or more of the patterns are illustratively depicted inconjunction with (e.g., over one or more cells) the VisualY modelframework as part of an instantiation of the VisualY language. Thoseskilled in the art will appreciate that, when the patterns areincorporated into an instantiation, the changing of the variables may beuser-initiated but some or all of them are also automatically adjustedprogrammatically. For example, the generative nature of the programmingof engine 170 will illustratively drive changes in the variables andtherefore changes in the patterns and therefore changes in theinstantiation. FIG. 12 is a screen shot of a user interface forselecting and thereby initiating display of an instantiation of theVisualY language.

Accordingly, embodiments of the present invention pertain not just todata visualization but to knowledge visualization. In one embodiment,all instantiations of the VisualY language are supported, meaningassumptions in the VisualY model framework and/or the VisualYrepresentations, by proofs that are may be accessed, for example but notby limitation, by navigating (e.g., navigating to a lowest level)through the VisualY model framework as described herein. The proofsillustratively provide source data to back up the assumptions in termsof how constructs are related and presented as being connected.

In one embodiment, relationships such as but not limited to datarelationships are embedded in the coding of the VisualY language. Thus,the system is generative at least in that the applications engines usedata to determine, for example based on first principles as describedherein, what the programmatic response of the VisualY instantiation willbe. In essence, the behavior of a manifestation of the VisualY languageover time is illustratively generative in that it is data dependent—andthose dependencies are designed with the intent of being as consistentas possible with first principles of science and nature. Thedependencies are illustratively adjusted as necessary to be as accurateas possible. Thus, manifestations of the VisualY language are much morethan simple data visualization. The manifestations are such that data istruly relational in that it is built into the programmatic assumptionsof the coding.

Further, embodiments described herein go beyond known methods andsystems in the field of semiotics. Embodiments do involve semiotics inthe sense that they incorporate unique uses of signs, sign systems andsign processes, symbols, signification, systems of communication,meaning, language, biosemiotics, and computational semiotics. However,embodiments described herein also involve cognitive semiotics, andembodiments involve a semiotic coding system incorporating syntacticsand cybernetics. The systems in which knowledge is presented inaccordance with instantiations of the VisualY language are organic inthat they are dynamically generative and not necessarily predeterminedor static. Embodiments are robust and scalable.

In one embodiment, the VisualY language is organized around an objectivedriven concept. What the user sees in an instantiation of the VisualYlanguage is what they want. The user essentially makes selections andnavigations so as to adjust the instantiation and effectively filter theuser interface down to what they want.

The spirit of the VisualY language is rooted in the study of semioticsat least in that it prescribes multimedia devices as core components ofa system for communicating representations of complex knowledge.Standardized signs, symbols, colors, patterns, motions and sounds areset against a standardized multi-level ontological framework of maps andmap views. Standardized tools (illustratively but not necessarilyInternet browser based tools) are provided to enable a distributedcommunity of users to interact with and/or enhance the multimediarepresentations and/or the framework. Thus, members of the usercommunity operate in a wiki-style format to communicate with one anotherusing the language of VisualY.

FIG. 13 is an illustrative screenshot representation of a limitedportion 1300 of an embodiment of the VisualY framework. Portion 1300illustratively represents a person that is the focus of an activeinstantiation of the VisualY language. Portion 1300 is illustratively anexpanded view of the circled “Y” human focus area 326 of the “forms ofexistence” framework scheme shown in FIG. 3.

In order to emphasize the concept of variable segmentation as itpertains to embodiments of the present invention, the portion 1300 shownin FIG. 13 has many more cells than the Yperson representation in area326 of FIG. 3. With that in mind, the representation 1300 and therepresentation in area 326 are illustratively alternative VisualYframework representations of the same thing, the differences between thetwo representations being attributable to the state of a user-adjustablemechanism for choosing a desired level of displayed detail. The usercan, directly or indirectly, intentionally or unintentionally, increaseor decrease the level of detail by manipulating a user control providedwithin the application user interfaces.

In one embodiment, the representation in FIG. 13 is what the user seesafter they increase the size of the Yperson representation in area 326of FIG. 3. For example, but not by limitation, the user may do this byeffectuating a pinching gesture wherein two of the user's fingers areplaced against a touchscreen on the Yperson representation in area 326,and then the fingers are slid away from one another in a manner similarto how many photo applications enable a user to increase the size of animage such as a photograph. By performing and perhaps repeating thisgesture, the user is illustratively able to increase the size of theYperson representation until it looks similar to what is shown in FIG.13. Obviously, the enlarged representation has more framework cells thanits smaller counterpart. In one embodiment, as the size is increased,more cells begin to pop into the Yperson representation. These cells areillustratively swept into the larger image from other underlying levelsof the framework.

The framework representation 1300 in FIG. 13 is shown without anyindication of other portions of the VisualY framework outside of thecircled “y” portion. However, in one embodiment, representation 1300 issimplified in that other portions of the framework are likely to berepresented together with the circled “y” portion 1300. For example,representations of cells on other levels of the VisualY framework may beincluded. Also or alternatively, other framework portions and cells fromthe same framework level such as but not limited to any of the othercells in areas 308 and 310 of the FIG. 3 framework representation. Asthe size of the Yperson representation in area 326 of FIG. 3 isincreased, other portions of the framework need not be eliminated tomake room but may instead be shrunken or otherwise simplified (e.g., asmaller short-hand representation, or a more encompassing collectiverepresentation) to accommodate the transition to a more detailedrepresentation of the circled “Y” portion of the framework, such as thatshown in FIG. 13.

What is not really represented in either of FIG. 3 or 13 is anysignificant signs, symbols, colors, patterns, motions or sounds providedas part of a VisualY representation in accordance with an instantiationof the VisualY language. FIGS. 14 and 15 are illustrative screenshotrepresentations of an active VisualY representation being depicted, aspart of an active instantiation of the Visual Y language, in relation toan expanded view of the circled “Y” human focus area 326 of the “formsof existence” framework scheme shown in FIG. 3. It is to be understoodthat FIGS. 14 and 15 show non-moving black and white representations butare actually likely to include color, motion, and even sound.

The VisualY representation 1400 of FIG. 14 includes a plurality ofmoving, semi-transparent shards 1402 (only a representative few havebeen labeled). Each shard 1402 illustratively is connected at somelocation to at least one segment or cell included in the underlyingportion of the VisualY framework. The shards thereby represent activitywithin that segment or cell of the framework.

In areas of representation 1400 such as area 1404 where multiple shardsare bunched together, this can mean that multiple shards are connectedto the same segment or cell. Also or alternatively, it can mean that thecurrent user-selected level of framework detail is not “zoomed out”enough to enable a more clear distinction between shards and theirassociated segments or cells. In one embodiment, the user can adjust theview of the VisualY framework so as to change the depicted perspectiveof a single segment/cell or a group of segments/cells and therebyrelieve the congestion of shards.

Shards 1402 are programmatically configured to convey meaning andrelationships between segments/cells through mechanisms such as but notlimited to their color, patterns, movement and/or associated sounds. Inone embodiment, to the extent possible, the meaning and relationshipsconveyed by the shards 1402 are established and adjusted automaticallyor manually by users of the application (such as by gatekeeper 120 orusers 110 through user N shown in FIG. 2) in order to maintain as muchconformance as possible with first principles of science.

The VisualY representation 1500 of FIG. 15 includes a plurality ofmoving dots 1502 (only a representative few have been labeled). Each dot1502 illustratively is connected at some location to at least onesegment or cell included in the underlying portion of the VisualYframework. The shards thereby represent activity within that segment orcell of the framework. Within FIG. 15 the boundaries of the underlyingsegments/cells of the VisualY framework are not specifically shown otherthan where there is a relative absence of dots forming a capital “Y”shape. In one embodiment, a clearer depiction of one or moresegments/sells can be selectively (e.g., in accordance with auser-actuated control) or automatically added to the representation1500.

In areas of representation 1500 such as area where multiple dots 1502are bunched together, this can mean that multiple dots are activerelative to the same segment or cell of the VisualY framework. Also oralternatively, it can mean that the current user-selected level offramework detail is not “zoomed out” enough to enable a more cleardistinction between dots 1502 and their corresponding segments or cells.In one embodiment, the user can adjust the view of the VisualY frameworkso as to change the depicted perspective of a single segment/cell or agroup of segments/cells and thereby relieve the congestion of shards.

Dots 1502 are programmatically configured to convey meaning andrelationships between segments/cells through mechanisms such as but notlimited to their color, patterns, movement and/or associated sounds. Inone embodiment, to the extent possible, the meaning and relationshipsconveyed by the dots 1502 are established and adjusted automatically ormanually by users of the application (such as by gatekeeper 120 or users110 through user N shown in FIG. 2) in order to maintain as muchconformance as possible with first principles of science.

Accordingly, it is within the scope of the present invention for VisualYrepresentations to demonstrate knowledge in many ways such as but notlimited to patterns of relationships between multiple variables (i.e.,between multiple cells or segments) utilizing mechanisms other than avisual line or other similar connecter joining related elements.Utilization of a line or other similar connector is less effectiveapproach to demonstrate relatedness at least because it is a veryinefficient use of display space. Further, human beings have a difficulttime following and deriving meaning from more than a couple of lines ata time. A representation with many, many lines makes it very difficultto identify pattern differentiators within complex systems, and thesedifferentiators can be indicative of especially interesting knowledge.

As opposed to a static image filled with many lines and nodes,instantiations of the VisualY language are capable of depictingknowledge patterns across numerous dynamically adjustable variants anddimensions. The instantiations may be selectively altered by a user toaccommodate any special circumstance, perspective, variable, timecontext, or view. The user is able to make adjustments to variablesapplicable to the VisualY representation and/or the VisualY modelframework, and then experience the effects of those adjustments on theinstantiation, and therefore the effects of those adjustments in actualexistence because the effects are programmatically determined based onfirst principles of science. VisualY language instantiations areadaptive, scalable, and not limited to a specific domain or context.

Further, instantiations of the VisualY language enable users to easilysee and understand knowledge across many variables because patterns areeasy to differentiate across the broad and encompassing VisualYframework. The user sees connections between variables (e.g., betweensegments and cells of the VisualY framework) from an overall systemperspective but still across many different levels of nature andexistence.

In one embodiment, VisualY representations are also programmaticallyconfigurable to take into consideration an objective input by a user ofthe application. In this case colors, motions, patterns, sounds andother VisualY representation mechanisms are presented in the context ofthe VisualY model framework so as to programmatically account for theinput objective. This makes it even easier for the user to pick up onimportant knowledge in the form of, for example, patterns among thedifferent segments/cells because the VisualY instantiation and the userare on the “same page” in terms of understanding the objective contextof the instantiation. Users can illustratively choose any type ofobjective such as, but certainly not limited to, to learn about aparticular topic, to shop for a particular item, to make a decision, orto find a particular place.

It is worth pointing out that, as broad and encompassing asinstantiations of the VisualY language may be, there are limitationsincorporated into in the representative scheme in order to impose apredictable level of contextual predictability and standardization.These limitations are the tools utilized to communicate thedifferentiating patterns that communicate knowledge. The VisualY modelframework provides a pre-built model framework and associated fixedviews that users of the application can depend upon to remain consistentin format. Even the VisualY representations include some degree ofpredictability such as a limited color scheme and a limited number ofvisual mechanisms for showing connections between variables. Thestandardized elements of the VisualY schemes are the tools with whichcomplex knowledge is communicated as an instantiation of the VisualYlanguage.

With all of this in mind, one can imagine a user experiencing aninstantiation of the VisualY language. For example, the user willillustratively see where VisualY representation activity within theVisualY model framework is the greatest, perhaps after communicating andcausing a particular objective to be added to the calculus of thepresented patterns. The user will see currents across segments/cellswithin the model framework demonstrating movement across differentdimensions and/or relationships between different segments/cells. Forexample, related segments/cells may have the same moving cross-hatchpattern, which may or may not be the same color. Different colors willsignal something about the relationship to the user and that somethingwill be understood because that type of differentiation will be utilizedin the same way consistently in the VisualY scheme to signify the samerelationship characteristic. Finally, the user will illustratively beable to activate and deactivate different user input controls in orderto change filters and thereby change the current instantiation of theVisualY language to a different but possibly related instantiation ofthe VisualY language. The user can illustratively change whichsegments/cells are larger or smaller in order to gain a more clearunderstanding of how the VisualY representation interrelates differentvariables. The use is in control. The user gets to choose theirobjective and also chose how they experience what it is that they areseeing. By navigating around through the model, making adjustments tovariables as desired, and by changing how information is presented, theuser is able to look for important patterns that are indicative ofknowledge and/or are indicative of what it is that the user is searchingfor in terms of their communicated objective. Thus, the user drives thedata visualization processes looking for patterns in color, angles,flows, motion, sound and the like in the context of the model frameworkand its understood components.

Instantiations of the VisualY language are an excellent device forclearly communicating knowledge involving simultaneous actions. Manyknown devices for communicating knowledge are extremely linear and onlyaddress one action at a time. It is left to the individual to derive anunderstanding as to how multiple, separate actions coincide and areinterrelated. Some devices may provide insight into theinterrelationship between a couple of actions but attempts to movebeyond a couple of actions quickly become muddled with information thatmake it difficult if not impossible for humans to clearly understand.

For example, to effectively understand the economic situation of a givencountry, one should consider active interrelationships between howdifferent categories of citizens feel, business performance, the effectsof the influence of other countries, changes in political leadership,etc., etc., etc. An instantiation of the VisualY language enables a userto experience the effects of a tremendous number of variables at thesame time. The human mind is able to process the colors, motion,patterns, symbols, sounds and other mechanisms of a VisualYrepresentation in a much more efficient and sophisticated way than itcan derive meaning from complex simultaneous actions presented incharts, graphs, language, and even traditional computer simulation.Further, a user can easily navigate through different levels andperspectives of the VisualY framework in order to increase or decreasethe complexity of the context within which the VisualY representation ispresented, thereby enabling a customized focus on what is of particularinterest to the user. Further, the user can selectively alter variablesan experience how the impact of that alteration on the many othervariables in the system. The results of the alteration may beunexpected, though still rooted in the first principles of science uponwhich interactions and behaviors of VisualY representations areprogrammatically bound.

An instantiation of the VisualY language is also especially effective atproviding an understanding of how different actions play out relative toeach other across time. The user is able to selectively determine a timeperiod over which elements of VisualY representations move and change,thereby indicating what is happening on many different levels ofexistence over that time period. Sticking with the example of a userinterested in the economy of a given country, the user can, for exampleby using a user interface mechanism to assign a specific value or valuesto one or more of the VisualY framework cells, set the VisualYrepresentation to the country of interest and to the economic status ofthat country. Then, the user can similarly provide inputs that willcause a VisualY representation of the status of the economy of theselected country to be shown at a certain rate over a certain time. Forexample, the user might request that the representation encompass a 100year period at one year per minute, or a 1 year period over one monthper minute, etc. As events such as war affect the economy of the countryin the actual events of history those events will be represented in thecolors, motions, sounds, symbols, cell interrelationships and othermechanisms of the VisualY representation. Thus, the user is able toexperience the effects of many different variables over time. Again, theuser is in charge of what they want to see in a given instantiation ofthe VisualY language and how they want to see it.

An instantiation of the VisualY language is also especially effective atproviding an understanding of how different actions play out relative toeach other across space. The user is able to selectively determine, forexample by setting an applicable instantiation view or representationparameter when space (e.g., geographic or otherwise) is to be accountedfor in a given VisualY representation. Thus, an instantiation of theVisualY language enables a user to observe patterns of existence acrossspace. For example, the user can set the application to for theobjective learning about how a clothing article such as a pair of pantscame to exist over time and space. This could mean VisualY elementsmoving and lighting up in relation to indications in the VisualYframework of different geographic regions in the world where differentmaterials in the pants originated. However, rather than simply beingtightly focused on the broad concept of the material itself, manyrelated variables related to the production of the materials such aslabor considerations, national economic considerations and any other ofother variables will be accounted for in the instantiation and broughtto the user's attention in the form of the colors, motions, sounds andother mechanisms of conveying information as part of the VisualYrepresentation. If the user becomes interested in any particular elementof the representation, here or she can manipulate the VisualY frameworkcontext and/or the views or configuration of the VisualY representationsettings as desired to focus on what is of interest at the moment.

Accordingly, instantiations of the VisualY language go much deeper thansimply data visualization. Data visualization is far more limited interms of being able to clearly communicate interrelationships among manychanging variables. Unfortunately, humans are used to attempting toanswer questions within the grasps of such constraints. Instead theinstantiations of the present invention are vehicles for communicatingmuch deeper knowledge and answering questions in a far more encompassingand accurate way. For example, a question like whether coffee is bad foryou is complex and dependent upon a multitude of variables such as butnot limited to everything else the person is eating and drinking, theage of the person, their gender, what is in the coffee, what kind ofcoffee is it, where did the coffee come from, what is in the water usedto make the coffee, where did the coffee come from, what is the price ofthe coffee, where did it come from, who harvested the coffee and howwere they compensated, etc. There are many potential variables toconsider and an instantiation of the VisualY language is an effectivetool for considering more than the interplay of just a couple of thevariables.

The fact of the matter is that there is often a gap between what isreally happening and what is perceived or understood. An instantiationof the VisualY language enables a conveyance of what is actuallyhappening, for example in accordance with first principles of science.Using language, charts, bars, graphs, and the like it is extremelydifficult to communicate, for example, the interplay of 24 differentvariables that contribute to an action that, on the surface, seems muchsimpler. For example, when you ask a person how they are feeling after along run, they might simply report being tired or a little bit sore. Butan instantiation of the VisualY language will demonstrate the actualphysical map the body including information such as where muscles aretorn or stretched, where stress is having an impact, where the body isnot functioning properly, etc. The mechanisms of a VisualYrepresentation will cause color, motion, sound, patterns and the like tobring many portions of the VisualY framework to an active state thatwill immediately convey deep knowledge that is otherwise ignored oroverlooked.

One can imagine a scenario in which a user inputs a particular objectiveinto the system, specifically to learn about head injuries in generaland concussions in particular. Referring to the Yperson frameworkportion 1300 shown in FIG. 13, there are illustratively different pointsof entry into the Yperson. At least one of the framework segments/cellsin the bottom right-hand area of the “Y” model between lines 1304 and1302 illustratively is designated as being a point where an absorptionof kinetic energy occurs as part of an exchange of kinetic energy. Thatcell illustratively lights up in the color red to show the kineticenergy coming into the Yperson.

As an aside, the color red is illustratively part of a limited spectrumof colors that are permitted within instantiations of the VisualYlanguage. For example, in one embodiment, two different shades of redare utilized to signify varying degrees of bad or degrees of generalnegative connotation. The darker of the two reds illustrativelysignifies more bad than the lighter shade. Two different shades of greenare illustratively utilized varying degrees of good or degrees ofgeneral positive connotations. The darker of the two greensillustratively signifies more good than the lighter shade. Then, thereis illustratively a color between the good and bad shades to indicate aneutral connotation—as in neither bad nor good.

Continuing with the concussion example, the kinetic energy entry pointin the bottom right-hand portion of the Yperson 1300 illustrativelylights up with the a shard or dot or some other pattern to indicate thekinetic energy coming into the Yperson. This is the beginning of theVisualY representation. As the representation continues, the patternillustratively shows transference of the red to other cells in thebottom right-hand portion. For example, other cells illustratively lightup so as to indicate blunt force damage to the brain in general and morespecifically to certain parts of the brain. The color significantcomponents of the VisualY representation continue to move from cell tocell in accordance with first principles of science as imposed by theengine described above in other sections. Relevant connections betweenthe various segments/cells are shown in any of a variety of differentways during the VisualY representation such as a moving flow of theshards, dots or a similar image elements from cell to cell. Also oralternatively, similarly configured moving or static cross-hatching isincorporated into separate but related cells. When two cells have thesame cross-hatch pattern and/or a cross-hatch pattern with flowingmotion in a similar direction, an inference of interrelatedness can beassumed. As will become apparent, in conjunction with the propagation ofthe kinetic energy from cell to cell in the bottom right hand portion ofthe Yperson, additional VisualY cell activity in other conscious andsubconscious regions of the Yperson will activate and begin their ownprogrammatic patterns in response to the event.

The described blunt force entry of kinetic energy into the Ypersonillustratively is depicted within the VisualY representation as a flareof energy that comes from outside of the VisualY framework portion 1300of FIG. 13 and enters at the kinetic energy absorption point in thebottom right hand region. There may be an indication of the object withwhich the Yperson has collided when the blunt force occurred. In oneembodiment, the playing out of the VisualY representation includes acorresponding coordinated video in an area of the VisualY framework suchas area 366 in FIG. 3. The video might, for example, show an image of aperson (i.e., the Yperson) being struck in the head by a hockey puck.The activity occurring within the Yperson framework 1300 and otherportions of the VisualY framework is illustratively timed with the videoin terms of what is happening and when. The timing is illustratively asconsistent with first principles of science as possible, which isprogrammatically accounted for in the logic carried out by the VisualYengines.

At the same time as the effects of the collision are being shown withinthe Yperson 1300, the effects are also being accounted for in terms ofsimultaneously relevant activity in other relevant portions of theVisualY framework. For example, cells are progressively lit up in area310 to show what is occurring at the molecular and other levels ofexistence. It is important to understand that the depiction of what isoccurring as a result of the collisions is not limited to only theimpact with the Yperson context.

Following the point of impact in the lower right hand portion of theYperson, the VisualY representation activity illustratively radiates toadditional relevant segments/cells eventually more toward the center ofthe circle. This is not arbitrary but is consistent with theprogrammatic scientific considerations that the impact will affect areashaving to do with processing. The effect of the impact will generallyshow up in the segments of more micro scoped human functions andconsiderations but then will permeate into macro scoped functions andconsiderations. In one embodiment, the impact will cause a flare thatpulses and then stays to some extent in certain segments/cells as timegoes on. Again, this is not arbitrary but is consistent with the firstprinciples of science upon which the behaviors of the VirtualYrepresentations are based.

The concussion representation activity illustratively moves through thehuman processing ring segments and propagates through molecules, etc.Impacted human cells are reflected as being affected, for exampleactivity occurs in certain framework segments to represent an impact onsurface cells, as well as deformed and possibly damaged brain cells. Oneafter another, relevant framework cells will light up with activity,often with red areas of activity to indicate a negative impact.Framework segments/cells will activate when a change in the heart (e.g.,change in beat pattern) or other organs occur. Activation occurscertainly to show different effects on the nervous system.

Some cells may present neutral activity. For example, the Yperson'spredisposition to negative concussion effects may be factored into whatoccurs. The predisposition may be a variable that can be adjusted by theuser before and after starting or re-starting the presentation of whathappens as a result of the impact.

The effects of the impact will, when the time is right, propagate intothe upper segments/cells in the top of the Yperson representation. Thisis where conscious considerations are accounted for. The framework cellsthat light up in the top area demonstrate when the touch of the impactis consciously observed. This is where the cognitive repercussions ofthe impact will be depicted. The bottom left side of the yperson iswhere the framework cells related to subconscious considerations areaccounted for in the activated cells of the VisualY representation.

Accordingly, the effects of the impact will propagate into all areas ofthe Yperson where, according to first principles of science, there isany sort of impact. The person interested in concussions will becomeaware of relevant considerations not just on all levels of the Ypersonbut on all levels represented in the larger VisualY framework. The usercan interact with application input mechanisms to start, pause, andre-start the impact and the events that occur before or after it. Theuser can, at any time, similarly interact with input mechanisms so as tochange framework views, change the number and size of segments and cellsin certain areas of the framework, change parameters or variablesaccounted for in what happens in accordance with the applicable firstprinciples, etc. The user can explore the effects of the impact on alldifferent levels, and can see the simultaneous effects of the impact onlots of different levels. The concussion-focused instantiation of theVisualY language indicates all relevant exchanges and states from thosepertaining to electrical currents to those pertaining to chemicalreactions, as well as hormonal changes, social considerations, emotions,feelings, etc. The user can modify variables such as different treatmentoptions to see the effects of the treatments on the many levels of whathas and is being represented.

It is worth elaborating upon the “language” part of the VisualYlanguage. In accordance with another aspect of the present invention,the VisualY language is a language at least because sophisticated visualand auditory components that humans are especially skilled at areselected for a particular manifestation based on the meaning that theyconvey. In other words, a manifestation of the VisualY language is aselected and specially configured set of the sophisticated visual andauditory components. The selection and configuration of the set ofcomponents is done based on standardized meaning assigned to eachcomponent.

The VisualY language includes a many different attributes of visuallyperceivable movements, each attribute of movement potentially having adifferent meaning assigned or otherwise attached to it. Fast movement,for example, illustratively may mean one thing (e.g., but not bylimitation, urgency or rapid change), while slow movement may meananother (e.g., but not by limitation non-urgency or slow change). Ofcourse, attributes of movement to which meaning is attached inaccordance with the VisualY language are not limited to one attributeextreme or the other. An attribute or characteristic of movement towhich meaning is assigned may fall as a matter of degree anywhere on acontinuous spectrum from one extreme to the other, the relative positionon the spectrum potentially communicating an assigned meaning in and ofitself.

Meaning within the language of VisualY is attachable to any attribute ofmoment, no matter how subtle. This is an especially interesting part ofthe present invention because the human brain is particularly wellsuited for perceiving even subtle attributes of movement. For example,meaning can be assigned to the direction of movement, to the relation ofmovement in terms of proximity to other visual components, to thesmoothness or non-smoothness of movement, to relativity of multiplemoving components, to direction of movement, to a pattern of movement,to amplitude of movement, to expansiveness of movement, to changes in apattern of movement, to a relationship between movement and sound, totexture of movement, to a relationship to what in particular is moving,to what is created as the movement occurs, to changes caused by themovement, etc., etc., etc. Of course, these are but examples ofattributes of movement to which meaning may be attached as part of theVisualY language. The scope of the present invention is certainly notlimited to these examples. Thus, instantiations of the VisualY languageconvey layers upon layers of meaning through the movement of visualcomponents, and the human brain is well suited to pick up much more ofsuch meaning than is possible when taking in meaning from a spokenlanguage.

What exactly it is that is moving may also play a role in the meaningthat is conveyed as part of an instantiation of the VisualY language.For example, what it is that moves may be lines, integrated groupings ofcolors, patterns, etc. The component that is moving itselfillustratively has assigned meaning and the movement adds to thatmeaning. Instantiations of the VisualY language reflect the energy ofthe way things move and interact in reality (or reality as it has beendefined in the context of the particular instantiation). This comes asno surprise because the programmatic logic that determines how movementis allowed to be presented is illustratively based on first principlesof science applied by the engines that have already been described inother sections of the present invention. Thus, by watching aninstantiation of the VisualY language, the user becomes aware of theenergy and interactions of what it is that they are watching. Shape,color, form, texture and many other attributes collectively communicatean incredible amount of information to the user more at less all at once(though the actual instantiation may play out over a period of time).The flow, the relative positioning of components, movement, color andall other human perceivable attributes of a Visualy representation cometogether to capture knowledge and present it to the user in a veryeffective and efficient manner.

It is within the scope of the present invention for any observableelement detectable by the human senses (primarily hearing, seeing,touching) and presentable through an application user interface (e.g., adisplay such as a computer screen, sound from speakers, tactile feedbackprovided by way of a touch sensitive output mechanism, etc.) to beassigned some level of meaning and thereby be made a part of the VisualYlanguage. As part of the VisualY language, the element is selectivelyincorporated into an instantiation of the language so as to contributeits assigned meaning to the instantiation in combination with otherVisualY elements contributing their own meanings so as to collectivelyconvey information and, in particular, knowledge. Thus, the VisualYlanguage is flexible and very effectively informative, especially whenit comes to conveying the simultaneous and intertwined impact ofmultiple system variables, and especially as compared to other types oflanguages used to communicate knowledge and information, such as thespoken languages.

Additional examples of observable elements detectable by the humansenses to which meaning may be assigned and thereby be made part of theVisualY language will now be provided. Color may be assigned meaning andmade part of the VisualY language. For example, in one embodiment, theVisualY language includes a limited spectrum of colors with colors onone side of the spectrum representing varying degrees of “good,” withcolors on the opposite side of the spectrum representing varying degreesof “bad,” and with one or more colors in the middle representing a“neutral” state that is neither good nor bad. When “good” colors arepresented within an instantiation of the VisualY language, the userillustratively infers a good context such as happiness, pleasure orpositive growth. The opposite is true for “bad” colors, which mighttrigger an inference of damage, pain or destruction. Of course, it isalso within the scope of the present invention for the meaning of aparticular color to very depending upon a contextual characteristicrelevant to a particular VisualY language instantiation, such as aparticular VisualY model framework context.

It is not just static or isolated representations of color that may beassigned meaning and thereby be made part of the VisualY language.Transitions between colors, combinations of colors, the shape of avisual element that includes a given color, color patterns, theproximity of color to other VisualY language elements, or any otherattribute of color may be assigned meaning and made part of the VisualYlanguage. Further, any attribute of color such as shade, brightness,coolness, warmness, hue, saturation, texture, shadow, transparency,opaqueness, luminosity, cultural considerations, popular dispositions orconnotations, psychological or emotional variables, physiologicalvariables, environmental variables, variables related to achromaticsimultaneous contrast properties, variables related to metamerismproperties, or any other properties or variables may be assigned meaningand made part of the VisualY language.

Sounds may be assigned meaning and made part of the VisualY language.For example, in one embodiment, the VisualY language includes a limitedspectrum of tones with tones on one side of the spectrum signifyingvarying degrees of the occurrence of rapid change, with tones on theopposite side of the spectrum signifying varying degrees of theoccurrence of slow change, and with one or more tones in the middlerepresenting a “neutral” state where there is little or no changeoccurring at all. Of course, it is also within the scope of the presentinvention for the meaning of a particular sound to very depending upon acontextual characteristic relevant to a particular VisualY languageinstantiation, such as a particular VisualY model framework context.

It is not just static or isolated sounds that may be assigned meaningand thereby be made part of the VisualY language. Transitions betweensounds, combinations of sounds, visual components with which sound isaffiliated, the apparent source of sound, the shape of sound, soundpatterns, the proximity of sound to other VisualY language elements, orany other attribute of sound may be assigned meaning and made part ofthe VisualY language. Further, any attribute of sound such as the speed,frequency, loudness, coolness, warmness, perceived abrasiveness,cultural considerations, popular dispositions or connotations,psychological or emotional variables, physiological variables,environmental variables, or any other properties or variables may beassigned meaning and made part of the VisualY language.

In accordance with another aspect of the present invention, aninstantiation of the VisualY language also includes one or more related“proofs.” Users are able to selectively access those proofs by way ofuser interface proof navigation elements. A “proof,” as that term isused herein, is a record, illustratively a visual record of evidence,hypothesis, logic, reason, validity, assumptions, intent, accuracy,systematic completeness, or another consideration reflected in theVisualY representation and or VisualY framework of the associatedinstantiation of the VisualY language.

A proof for an instantiation of the VisualY language is illustrativelyitself an instantiation of a proof model. In other words, there is aproof model that includes a variety of different proof elements andproof element details that are selectively combined (e.g., by way ofuser input, programmatic automated input, etc.) so as to form aninstantiation of the proof model for a given instantiation of theVisualY model. In one embodiment, the proof model includes triangularshapes that interconnect in a way that is indicative of derivedrelationships amount evidence, hypothesis, logic, reason, validity,assumptions, intent, accuracy, systemic completeness, etc.

Embodiments of the present invention provide a new approach to resolvingthe meaning of everything, the workings of everything, the knowledge ofeverything and the complex code and interactions of everything. Whilemuch of the present description has been focused upon the visual andaudio delivering methods for conveying patterns of meaning, it is worthelaborating upon the types of meaning that can be conveyed usingembodiments of the present invention. Below is a non-exhaustive list ofthe first principles meaning and knowledge that may be conveyed bypattern and form in accordance with aspects of the present invention. Inaccordance with aspects of the present invention, dynamic generativepatterns applied to models can directly convey all forms of systemicmeaning and knowledge, including:

Cause/Effect Variables Involved Input/Reactor/Output Add/SubtractAttract/Repel Reproduce/Replicate/Generate/ReGenerate/DeGenerate

Change: time, space, proximity, state, condition, energy,matter/material, form

Exchanges/Transformations/Interactions Core Code/Genetics/ProgrammingTransducers/Catalysts/AmplifiersSensors/Activators/Switches/Controllers/Converters/Catalysts/Ignitors/ReactorsStates: Energy/Functionality/Optimization Behavior/Multiple SynchronousBehaviors/Multiple Asynchronous Behaviors/ConcurrentBehaviors/Sequential Behaviors

Movement: momentum, speed, directionality, flow, sequence, path

Conscious/Subconscious/Corporal Human ConstructsOrganization/Relationship/Hierarchy/Ontology/MultidimensionalRelationship Systemic Function Multidimensional Inter and IntraRelationships Relativity Proofs/Necessary Prerequisites/Validity/Chance

Building Blocks

What is claimed is:
 1. An interactive semiotic communication system, thesystem comprising: a software engine configured to allow a user torepresent and understand a complex system in a semiotic environment,wherein the semiotic environment comprises a multivariable system; atoolset comprising a semiotic ontological framework and accompanyingsemiotic representations; a graphical user interface, displayed on acomputing device with a display, configured to allow the user to accessand select an item from the toolset and otherwise interact with thecomplex system in the semiotic environment; and wherein the softwareengine is implemented on a computing device with a processor.
 2. Theinteractive semiotic communication system of claim 1, wherein thesemiotic representations comprise at least one of a selection ofsemiotic patterns, a selection of semiotic animations, a selection ofsemiotic sounds, a selection of semiotic objects or a selection ofsemiotic colors, wherein the user interface is configured to allow theuser to select and interact with one or more options from the toolset.3. The interactive semiotic communication system of claim 1, wherein thesoftware engine provides a generative user experience based at least inpart on the user's interaction with the multivariable system.
 4. Theinteractive semiotic communication system of claim 1, wherein thesoftware engine is configured to automatically update the semioticenvironment after a user selection of the item from the toolset.
 5. Theinteractive semiotic communication system of claim 4, wherein thesoftware engine is also configured to record and store the selection andcorresponding changes to the semiotic environment such that theselection and corresponding changes are available for later retrieval.6. The interactive semiotic communication system of claim 1, wherein thegraphical user interface allows a user to change a view of the semioticenvironment.
 7. The interactive semiotic communication system of claim1, wherein the software engine, through the computer with the processor,is configured to connect to a network and is further configured to,using the network connection, organize and communicate information.
 8. Amethod for exploring complexity in a semiotic environment, the methodcomprising: receiving, through an interactive portal, a selection of asemiotic ontological framework; receiving, through the interactiveportal, a selection of a complex system; receiving, through theinteractive portal, an indication of an objective; displaying theselected complex system, utilizing the selected semiotic ontologicalframework, on a display; altering the displayed complex system inresponse to a received setting configuration; and wherein the semioticenvironment is implemented on a computing device with a processor. 9.The method of claim 8, wherein the objective is learning.
 10. The methodof claim 8, wherein the objective is to make a decision.
 11. The methodof claim 8, wherein the interactive portal utilizes sensory elements todelineate portions of the complex system.
 12. The method of claim 8,wherein the received setting configuration is an input of newinformation.
 13. The method of claim 8, wherein the received settingconfiguration is a selection of a layer of the semiotic ontologicalframework.
 14. The method of claim 8, wherein the received settingconfiguration is a change in a view of a semiotic object within theselected semiotic ontological framework.
 15. A semiotic communicationsystem comprising: a user-configurable representation of reality,wherein the representation of reality is provided on a display andgenerated based at least in part on a series of settings input by auser; a toolset allowing the user to alter a criterion, wherein thecriterion at least in part contributes to the representation of reality,and wherein the toolset further comprises: a series of selectablesemiotic pattern schemes; and a series of selectable semioticontological frameworks; and a software engine, implemented on a computerwith a processor configured to receive the user-altered criterion andautomatically propagate the altered-criterion throughout therepresentation of reality.
 16. The semiotic communication system ofclaim 15, wherein altering a criterion comprises inputting a series ofhypothetical conditions.
 17. The semiotic communication system of claim15, and further comprising a series of selectable sensory elements. 18.The semiotic communication system of claim 15, and further comprising aseries of selectable motion sequences.
 19. The semiotic communicationsystem of claim 15, wherein the software engine through the computerwith the processor, is configured to connect to a network and is furtherconfigured to, using the network connection, organize and store agenerated representation of reality.
 20. The semiotic system of claim15, wherein the representation of reality is dynamic and responsive to areceived change in a user-set criterion.